Recovering from a Disaster in an Exchange Server 2007 Environment : Recovering from a Site Failure & Recovering from a Disk Failure

Recovering from a Site Failure

When a site becomes unavailable because of a physical access limitation or a disaster such as a fire or earthquake, steps must be taken to provide the recovery of the Exchange server in the site. Exchange does not have a single-step method of merging information from the failed site server into another server, so the process involves recovering the lost server in its entirety.

To prepare for the recovery of a failed site, an organization can create redundancy in a failover site. With redundancy built in to a remote site, the recovery and restore process can be minimized if a recovery needs to be performed.

For environments where SLAs offer very little time to bring up a recovery location, administrators should strongly consider implementing Cluster Continuous Replication, a new feature of Exchange 2007. 

Creating Redundant and Failover Sites

Redundant sites are created for a couple of different reasons. First, a redundant site can have a secondary Internet connection and bridgehead routing server so that if the primary site is down, the secondary site can be the focus for inbound and outbound email communications. This redundancy can be built, configured, and set to automatically provide failover in case of a site failure. 

The other reason for redundant site preparation is to provide a warm spare server site so that a company will be prepared to perform a restore of a site server in case of a site failure. The site recovery can simply be having server documentation available in another site or having a full image of server information stored in another site. The more preparatory work is conducted up front, the faster the organization will be able to recover from a system failure.

If you plan to utilize warm spares, be sure to update those warm spares with patches and applications as you apply them to the production systems. This eliminates several steps when it comes time to put them into use.

Creating the Failover Site

When an organization decides to plan for site failures as part of a disaster recovery solution, many areas need to be addressed and many options exist. For organizations looking for redundancy, network connectivity is a priority, along with spare servers that can accommodate the user load. The spare servers need to have enough disk space to accommodate a complete restore. As a best practice, to ensure a smooth transition, the following list of recommendations provides a starting point:

• Allocate the appropriate hardware devices, including servers with enough processing power and disk space to accommodate the restored machines’ resources.

• Host the organization’s DNS zones and records using primary DNS servers located at an Internet service provider (ISP) collocation facility, or have redundant DNS servers registered for the domain and located at both physical locations.

• Publish the recovery site’s IP address as a lower-priority MX record. This way, when the recovery server comes online you won’t have to wait for DNS propagation to advertise the new MX record.

• For the Exchange servers, ensure that the host records in the DNS tables are set to low Time to Live (TTL) values so that DNS changes do not take extended periods to propagate across the Internet. The Microsoft Windows Server 2003 default TTL is 1 hour.

• Ensure that network connectivity is already established and stable between sites and between each site and the Internet.

• Create at least one copy of backup tape medium for each site. One copy should remain at one location, and a second copy should be stored with an offsite data storage company. An optional third copy could be stored at another site location and can be used to restore the file to spare hardware on a regular basis, to restore Windows if a site failover is necessary.

• Have a copy of all disaster recovery documentation stored at multiple locations as well as at the offsite data storage company. This provides redundancy if a recovery becomes necessary.

Allocating hardware and making the site ready to act as a failover site are simple tasks in concept, but the actual failover and failback process can be troublesome. Keep in mind that the preceding list applies to failover sites, not mirrored or redundant sites configured to provide load balancing.

Failing Over Between Sites

Before failing over between sites can be successful, administrators need to be aware of what services need to failover and in which order of precedence. For example, before an Exchange server can be restored, Active Directory domain controllers, global catalog servers, and DNS servers must be available.

To keep such a cutover at a high level, the following tasks need to be executed in a timely manner:

Failing Back After Site Recovery

When the initial site is back online and available to handle client requests and provide access to data and networking services and applications, it is time to consider failing back the services. This can be a controversial subject because failback procedures are usually more difficult than the initial failover procedure. Most organizations plan on the failover and have a tested failover plan that might include database log shipping to the disaster recovery site. However, they do not plan how they can get the current data back to the restored servers in the main or preferred site.

Questions to consider for failing back are as follows:

• Will downtime be necessary to restore databases between the sites?

• When is the appropriate time to fail back?

• Is the failover site less functional than the preferred site? In other words, are only mission-critical services provided in the failover site, or is it a complete copy of the preferred site?

The answers really lie in the complexity of the failed-over environment. If the cutover is simple, there is no reason to wait to fail back.

Providing Alternative Methods of Client Connectivity

When failover sites are too expensive and are not an option, it does not mean that an organization cannot plan for site failures. Other lower-cost options are available but depend on how and where the employees do their work. For example, many times users who need to access email can do so without physically being at the site location. Email can be accessed remotely from other terminals or workstations.

The following are some ways to deal with these issues without renting or buying a separate failover site:

• Consider renting racks or cages at a local ISP to colocate servers that can be accessed during a site failure.

• Have users dial in from home to a terminal server hosted at an ISP to access Exchange.

• Set up remote user access using Terminal Services or Outlook Web Access at a redundant site so that users can access their email, calendar, and contacts from any location.

• Rent temporary office space, printers, networking equipment, and user workstations with common standard software packages such as Microsoft Office and Microsoft Internet Explorer. You can plan for and execute this option in about 1 day. If this is an option, be sure to find a computer rental agency first and get pricing before a failure occurs and you have no choice but to pay the rental rates.

Recovering from a Disk Failure

Organizations create disaster recovery plans and procedures to protect against a variety of system failures, but disk failures tend to be the most common in networking environments. The technology used to create processor chips and memory chips has improved drastically over the past couple decades, minimizing the failure of system boards. And although the quality of hard drives has also drastically improved over the years, because hard drives are constantly spinning, they have the most moving parts in a computer system and tend to be the items of most failure.

Key to a disk fault-tolerant solution is creating hardware fault tolerance on key server drives that can be recovered in case of failure. Information is stored on system, boot, and data volumes that have varying levels of recovery needs. Many options exist such as storage area networks (SANs) or various RAID levels to minimize the impact of drive failures.

Hardware-Based RAID Array Failure

Common uses of hardware-based disk arrays for Windows servers include RAID 1 (mirroring) for the operating system and RAID 5 (striped sets with parity) for separate data volumes. Some deployments use a single RAID-5 array for the OS, and data volumes for RAID 0+1 (mirrored striped sets) have been used in more recent deployments.

RAID controllers provide a firmware-based array-management interface, which can be accessed during system startup. This interface enables administrators to configure RAID controller options and manage disk arrays. This interface should be used to repair or reconfigure disk arrays if a problem or disk failure occurs.

Many controllers offer Windows-based applications that can be used to manage and create arrays. Of course, this requires the operating system to be started to access the Windows-based RAID controller application. Follow the manufacturer’s procedures on replacing a failed disk within hardware-based RAID arrays.

System Volume

If a system disk failure is encountered, the system can be left in a completely failed state. To prevent this problem from occurring, the administrator should always try to create the system disk on a fault-tolerant disk array such as RAID 1 or RAID 5. If the system disk was mirrored (RAID 1) in a hardware-based array, the operating system will operate and boot normally because the disk and partition referenced in the boot.ini file will remain the same and will be accessible. If the RAID-1 array was created within the operating system using Disk Manager or diskpart.exe, the mirrored disk can be accessed upon bootup by choosing the second option in the boot.ini file during startup. If a disk failure occurs on a software-based RAID-1 array during regular operation, no system disruption should be encountered.

Boot Volume

If Windows Server 2003 has been installed on the second or third partitions of a disk drive, a separate boot and system partition will be created. Most manufacturers require that for a system to boot up from a volume other than the primary partition, the partition must be marked active before functioning. To satisfy this requirement without having to change the active partition, Windows Server 2003 always tries to load the boot files on the first or active partition during installation, regardless of which partition or disk the system files will be loaded on. When this drive or volume fails, if the system volume is still intact, a boot disk can be used to boot into the OS and make the necessary modification after changing the drive.

Data Volume

A data volume is by far the simplest of all types of disks to recover. If an entire disk fails, simply replacing the disk, assigning the previously configured drive letter, and restoring the entire drive from backup will restore the data and permissions.

A few issues to watch out for include the following:

• Setting the correct permissions on the root of the drive

• Ensuring that file shares still work as desired

• Validating that data in the drive does not require a special restore procedure