BizTalk Server 2009 Operations : Disaster Recovery (part 2)

3. BizTalk Group Disaster Recovery Procedures

A disaster recovery event for the BizTalk Group consists of restoring the BizTalk Group databases as well as related non-BizTalk databases on the destination system SQL Server instances. This also includes any DTS packages as well as SQL Agent jobs that exist in the source system (production).

The first step is to ensure that the last backup set has been restored to all SQL Server instances that are part of the destination system. This can be confirmed by reviewing the Master.dbo.bts_LogShippingHistory table that is populated by the Get Backup History SQL Agent job. When a backup is successfully restored, the Restored column is set to 1, and the RestoreDateTime is set to the date/time the restore was completed. When all of the databases that are part of a backup set have been successfully restored, the backup set ID is written to the Master.dbo.bts_LogShippingLastRestoreSet table. Once you have confirmed that available backup files have been applied, follow these steps on each SQL Server instance in the destination system:

As promised, we next cover the disaster recovery procedures for the BizTalk runtime servers. Later subsections cover disaster recovery procedures for BAM and EDI functionality.

4. BizTalk Runtime Server Disaster Recovery Procedures

The BizTalk runtime servers in the destination system should have BizTalk Server 2009 as well as any required third-party adapters or software installed using the same guidelines for the production BizTalk runtime servers. There are generally two methods for setting up the BizTalk runtime servers:

• Method 1: Restore BizTalk Group, configure BizTalk servers in BizTalk Group, and deploy applications.

• Method 2: Configure disaster recovery BizTalk servers in production BizTalk Group, disable services, keep the server up to date, and run an update script to update locations of databases in the destination system.

Both methods have advantages and disadvantages, which you'll find out more about in our detailed discussion of these methods next.

4.1. Method 1

To proceed with method 1, first verify that procedures to restore the BizTalk Group databases and related application databases have been completed. Once completed, proceed with restoring the BizTalk runtime servers using method 1. Method 1 has all software preinstalled, but not configured, and without any applications deployed on the BizTalk servers in the destination system. When the BizTalk Group is restored in the destination system and the BizTalk severs are configured using Configuration.exe, select Join for the BizTalk Group, not Create. The first server configured should have the master secret restored on it and then designated as the master secret server for the BizTalk Group using the Enterprise SSO management tools. Once all of the BizTalk servers are configured in the BizTalk Group at the destination system, deploy the BizTalk applications (assemblies and bindings).

While many of the steps can be scripted, this method essentially brings online a new environment when recovering from a disaster. At the same time, it reduces the amount of ongoing maintenance work for the destination system to a degree, since just the latest version of the application is deployed.

4.2. Method 2

Method 2 also has all software preinstalled, but takes it a step further and actually configures the BizTalk servers in the destination system to be member servers in the production BizTalk Group. Applications (assemblies and bindings) are deployed to the destination system BizTalk servers just like in production, except that the BizTalk host instances and all other BizTalk-related Windows Services are disabled and do not perform any processing in the destination system. During a disaster recovery event, a script is run on the destination system BizTalk servers to update the new location of the BizTalk Group in the destination system SQL instances. Once updated, processing can be enabled. Method 2 is recommended because it results in a faster recovery and less change overall. To proceed with method 2, first verify that procedures to restore the BizTalk Group databases and related application databases have been completed.

Once verification is completed, perform these steps:

5. Restore Procedures for BAM

The BizTalk Server 2009 documentation covers these procedures extensively, so we won't repeat them here. BAM consists of SQL Server databases, SQL Analysis databases, and DTS packages. Refer to the section titled "Backing Up and Restoring BAM" in the BizTalk Server 2009 documentation for the details.

6. Other Disaster Recovery Tasks

This subsection covers other tasks and recommendations related to disaster recovery. Tracking data is an important part of a BizTalk solution, since that data can be used for reporting and as part of recordkeeping regulations compliance. It can also be used to help recover from a disaster, because it is a record of data processing activity. For this reason, we recommend separating your tracking databases from the runtime databases that generate tracking data by configuring your databases in separate SQL Server instances on different disks in production. Data in the tracking databases can be used to help determine the state of the system up to the point of failure for the runtime databases. Tracked messages and events can indicate what processes may have already happened and what messages have been received or sent.

The next subsection covers steps to evaluate data loss for the BizTalk Group with tips on how to recover data.

6.1. Evaluating Data Loss for the BizTalk Group

After data loss has occurred, recovering it is often difficult or impossible. For these reasons, using a fault-tolerant system to prevent data loss is extremely important. In any case, a disaster may occur, and even the most fault-tolerant system has some chance of failure. This subsection covers methods to help determine the state of the system when the failure occurred and how to evaluate corrective action.

6.1.1. Managing In-Flight Orchestrations

The Messagebox databases contain the state of orchestrations that are currently in progress. When data is lost from the Messagebox databases, it is not possible to tell exactly what data has been lost. Therefore, it will be necessary to examine external systems to see what activities have occurred in relation to the in-progress orchestrations.

Once it is determined what has occurred, steps can be taken to restore processes. For example, if upon looking at external systems or logs it is determined that an orchestration was activated but didn't perform any work, the message can be resubmitted to complete the operation.

It is important to consider what information will be available to compare with in-flight orchestrations in order to decide whether to terminate or resume particular in-flight orchestrations. Available information is largely determined by the architecture and design of the system such as what logging is performed "out-of-band" so as to not impact performance but at the same time provide an audit of events for comparison purposes.

6.1.2. Viewing After the Log Mark in Tracking Databases

While all databases need to be restored to the same mark for operational reasons in order to restore a consistent BizTalk Group, administrators can use a Tracking database that was not lost in Archive mode to see what happened after the mark. The process of evaluating the data begins by comparing services that are in flight in the BizTalk Administration Console Operations views against their state in BizTalk Group hub reporting. If the Group Hub Reporting shows it as having completed, the instances can be terminated.

BizTalk Message Tracking Reporting may show instances that started after the point of recovery. If so, any actions these instances took must be compensated, and then the initial activation messages can be submitted.

Reporting may also show that instances have progressed beyond the point at which the Operations view indicates. In this case, use the Orchestration Debugger in Reporting to see the last shapes that were executed, and then use Message Flow to see what message should have been sent or received. If they do not match the state in the Operations view, corrective action is required. Options are to terminate, compensate and restart, or resubmit any lost messages.

6.1.3. Marking In-Flight Transactions as Complete in BAM

BAM maintains data for incomplete trace instances in a special active instance table. If some instance records were started before the last backup but completed after the backup, those records will remain in the active instance table because the completion records for the instance will have been lost. Although this does not prevent the system from functioning, it may be desirable to mark these records as completed so that they can be moved out of the active instance table. To accomplish this, manual intervention is necessary.

A list of incomplete ActivityIDs for a given activity can be determined by issuing the following query against the BAM Primary Import database:

Select ActivityID from bam_<ActivityName> where IsComplete = 0

If data from external systems indicates that the activity instance is in fact completed, use the following query to manually complete the instance:

exec bam_<ActivityName>_PrimaryImport @ActivityID=N'<ActivityID>', @IsStartNew=0, @IsComplete=1

					  

7. Related Non-BizTalk Application Disaster Recovery Procedures

There may be additional non-BizTalk applications that must be restored as part of the overall application solution. If these application databases participate in distributed transactions with the BizTalk Group databases, the databases should be part of the Backup BizTalk Server SQL Agent job and restored to the same mark as the other BizTalk Group databases. In general, each individual application should have a disaster recovery plan tailored to the application that should be part of the overall solution disaster recovery plan.