BizTalk Server 2009 Operations : Maintaining the BizTalk Group (part 3) - Restore Procedures

3. Restore Procedures

Restoring a BizTalk-based solution, or any large application environment, is a challenging process if not well documented, tested, and periodically rehearsed from an operations training standpoint. BizTalk Server 2009 has its own set of unique steps required for successful restore. This subsection covers steps for recovering from various failure scenarios that do not require transitioning to the disaster recovery site. In some scenarios, this discussion leverages the destination system databases and the SQL Agent automation available in order to safely perform restore procedures minimizing manual steps. This discussion covers restore steps for the following scenarios:

• Restore procedures when refreshing the BizTalk databases in a development or test environment

• Restore procedures when migrating the production BizTalk databases to a more powerful production database server without switching to the disaster recovery site

• Restore procedures when recovering the production database environment from a hardware failure such as a SAN failure without switching to the disaster recovery site (destination system)

• Restore procedures for SQL Agent jobs and DTS packages

• Restore procedures for the master secret

• Restore procedures for the BizTalk runtime environment

The next subsection covers how to stop and start BizTalk processing while performing any of the restore scenarios listed previously.

3.1. Stopping and Starting BizTalk Application Processing

This subsection lists the steps to stop, start, pause, or resume BizTalk processing using either the Services administration tool or the command line. Depending on the features configured on a BizTalk runtime server, there are up to six services or more that must be managed:

• BizTalk Service BizTalk Group: <BizTalkServerApplication> (btssvc$BizTalkServerAppli cation)

• Enterprise Single Sign-On Service (Entsso)

• Rule Engine Update Service (ruleengineupdateservice)

• BAM Event Notification Service (NS$BamAlerts)

• World Wide Web Publishing Service (W3SVC)

Using the Services administration tool in the Control Panel, simply click Start, Stop, Pause, or Resume as needed for the scenario. From a command prompt, enter the following commands (where ServiceName corresponds to the values in parentheses in the preceding list):

• Start a service: net start ServiceName

• Stop a service: net stop ServiceName

• Pause a service: net pause ServiceName

• Resume a service: net continue ServiceName

These commands can of course be scripted by placing the commands in a batch file or using WMI so that all services can be controlled by a single step. For example, a Stop.cmd file could contain the following:

Net stop "edi subsystem"
Net stop Entsso
Net stop ruleengineupdatservice
Net stop btssvc$BizTalkServerApplication
Net stop NS$BamAlerts
Net stop w3svc

When writing scripts such as this, you may need multiple Net stop btssvc$BizTalkServerApplication lines depending on how many host instances are running on the BizTalk runtime server. Also, the services that are actually present on a BizTalk runtime server depend on the BizTalk configuration. For more information on the BizTalk services, go to the BizTalk Server 2009 core documentation and search for the topic titled "How to Start, Stop, Pause, Resume, or Restart BizTalk Server Services."

The next subsection covers restore procedures for refreshing a development or test environment.

3.2. Refreshing a Development or Test Environment

Here we cover the steps to restore a development environment or test environment back to a "known" or "initial state." These procedures can be used to help ensure that a testing environment remains consistent between test runs or as new functionality is added to the application.

This scenario requires a consistent full backup set of the following items:

• BizTalk databases

• Related non-BizTalk application databases

• Analysis Services databases for BAM or for Tracking Analysis Server

• DocumentsHome directory for EDI

• Windows SharePoint Services site customizations such as configuration files, site templates, client-side scripts, and so on

This procedure is based on "Moving BizTalk Server Databases" and related documentation in the BizTalk Server 2009 core documentation. Please be sure to review this section of the BizTalk documentation before proceeding. This is actually a simpler scenario than described in the documentation, because the BizTalk runtime servers and SQL Server database instance server names remain the same, so the steps to update database names and database locations via script are not required.

3.3. Migrating Production Databases to More Powerful Servers

Migration is probably the most common scenario that a BizTalk application operations team will encounter with respect to having to move a database. In this scenario, application processing is stopped using the procedure listed in the subsection titled "Stopping and Starting BizTalk Application Processing" earlier. The next step is to perform a full backup on the applicable database that is being migrated. Then perform the steps necessary to update references to the new database location for the BizTalk Group. The specific steps are covered in the BizTalk Server 2009 core documentation in the section titled "Moving BizTalk Server Databases" for the following databases:

• BAM Primary Import database

• BAM Archive database

• BAM Star Schema database

• BAM Analysis database

• BAM Notification Services database

• Messsagebox database

Please refer to the BizTalk Server 2009 core documentation for the specific steps to move these databases.

3.4. Recovering Production Database Environment from Hardware Failure

This scenario covers restoring the BizTalk Group databases that are backed up by the Backup BizTalk Server SQL Agent job. There are two options for this scenario.

Another option that leverages the disaster recovery infrastructure without actually performing a full disaster recovery is to obtain a copy of the databases from the disaster recovery destination system database instances and perform a restore of the databases at the production site. The second option is available if the BizTalk application operations team is able to correct the hardware failure in a timely manner in accordance with availability and corporate requirements and those server and database instance names do not change. 

The second option assumes that the BizTalk runtime servers, database servers, and all related servers and sites are back to fully operational, but the most up-to-date version of the application databases are required in order to restart processing. The disaster recovery site (destination system) will generally have the latest copy of the database available, so this scenario leverages the availability of those databases to quickly restore processing at the production site.

In order to obtain the latest version of the databases from the destination system, the operations team must ensure that all of the log backup sets (except for the latest set available) have been successfully restored to the databases. To determine the last successful backup set restored, review the contents of the Master.dbo.bts_LogShippingHistory table on each database instance in the destination system. This table is populated by the Get Backup History job and updated by the Restore Databases job. When a backup is successfully restored, the Restored column is set to 1, and the RestoredDateTime is set to the current date and time.

When all of the databases being restored to the server from a particular backup set have been successfully restored, that backup set ID is written to the Master.dbo.bts_LogShippingLastRestoreSet table. Once all of the log backup sets except for the last available have been confirmed to have been successfully restored, follow these steps:

3.5. SQL Agent Jobs

The SQL Agent jobs are a critical tool for maintaining the BizTalk Group. If one of these jobs is accidentally deleted or needs to be restored as part of restoring the BizTalk Group, obtain the backed-up script file for the SQL Agent job, and execute the script for the SQL Agent job in a query window. Check the status of the SQL Agent jobs in SQL Server. Ensure the correct account is configured to run each job.

3.6. DTS Packages

If the BizTalk solution has BAM configured with BM.exe, BM.exe will create a set of SSIS packages to produce the OLAP cubes. For each SSIS package, import the package using SQL Server 2005 Integration Services. Check the status of the DTS package in Enterprise Manager.

3.7. The Master Secret

The master secret is a critical item that must be backed up and stored in a safe place. It is not possible for BizTalk Server to function or to be restored without the master secret. There are two scenarios where the master secret server must be restored:

• Failure of the master secret server for an existing BizTalk Group

• Restoration of the master secret server as part of restoring a BizTalk Group to another set of servers

This subsection covers the procedures to restore the master secret server for the BizTalk Group. If the original master secret server fails and cannot be recovered but the BizTalk Group is still available and functioning except for the master secret server, another server can be promoted as the master secret server. To promote a Single Sign-On Server in the BizTalk Group to master secret server, follow these steps:

Now let's move on to the procedures for restoring the BizTalk runtime environment.

3.8. BizTalk Runtime Environment

If the BizTalk runtime environment is intact, then restoring the BizTalk runtime environment is a matter of verifying connectivity by opening the BizTalk Administration Console and then restarting processing by starting the items stopped in the subsection titled "Stopping and Starting BizTalk Application Processing" earlier.

If the BizTalk runtime environment must also be rebuilt and the server names will be the same, first install BizTalk following the same procedures originally used to install BizTalk.

Once BizTalk is installed, run Configuration.exe and join the BizTalk Group that has been previously restored.

Restore the master secret on the server with the same server name as the original master secret server. Or, restore the master secret to a new server and then perform the steps to designate the new server as the master secret server. Redeploy the BizTalk applications and bindings into the environment as needed.

Next, you'll learn the restore procedures for non-BizTalk, but related, applications and databases.

3.9. Related Non-BizTalk Applications and Databases

A BizTalk solution may update or receive data from non-BizTalk applications as part of the overall solution. If the related non-BizTalk applications include a database that participates in distributed transactions with BizTalk, this database should be added to the Backup BizTalk Server SQL Agent job and restored to the same log mark as the BizTalk databases. Otherwise, restore the non-BizTalk application and databases following the documentation for that application.