Microsoft Systems Management Server 2003 : The Four-Phase Patch Management Process (part 2) - The Evaluate & Plan Phase, The Deploy Phase

3. The Evaluate & Plan Phase

The Evaluate & Plan phase of the patch management process is where the organization needs to examine how it will respond to a software update, how it will release the update, how it will build the update, and how it will conduct acceptance testing for the release.

Determining the Appropriate Response

When an RFC is created, the initiator assigns an initial priority and category to the request. The patch management team should review the RFC and agree to, or change, the priority and category. The finally determined values will have an impact on the remainder of the patch management process, including how and when the software update is released. When the team reviews the initial priority attached to an RFC, it should consider what assets are impacted by the vulnerability the update addresses and whether these are critical systems, whether controls are in place (or can be put in place) to mitigate the vulnerability, and whether the update needs to be applied to as many systems as first thought. It’s recommended that the organization define priorities for release of software updates and time frames consistent with their needs, such as those in Table 2.

The organization might wish to consider formalizing the criteria used when determining whether or not to adjust a release’s priority. You can find an example of formalized criteria in the form of environmental and organizational factors and the corresponding priority adjustments in Table 3.

Emergency change requests, where vulnerability is being exploited within the production environment or system instability is affecting line-of-business applications, need to be handled expeditiously. These requests might cause other requests with lower priorities to be delayed or halted if already in deployment in order to free the necessary resources required to process them.

Not all software updates are the same, and each might have a different effect when applied to a system. Some updates might require a system to be rebooted while others do not. It’s possible that some updates will rely on other updates, perhaps a service pack, and can’t be installed without them. Also, some software updates, once applied, can’t be removed from a system. The patch management team members need to assess the impact that a software update will have on the environment in order to plan for it. This is called categorizing the update.

Once the release priority and category of the software update has been determined, the release needs to be reviewed and authorized for deployment. Before authorization is granted, the patch management team will need to consider various factors, such as what is currently happening in the production environment, the release’s projected cost, the best means of distributing the release, the resources required to manage the release, and any dependencies the release requires. Once authorization has been granted, a member of the team should take ownership of the release. The release owner is responsible for assembling the necessary resources to guarantee the building of the release, its testing, and its eventual deployment.

Planning the Release

Once the release has been approved, detailed planning should take place to guarantee the deployment’s success. Although considered previously in the Identify phase and when determining the appropriate response in this phase, the IT assets that need to receive the software update need to be identified and recorded. The Software Updates component of SMS 2003 can help identify assets that need updating, but if SMS 2003 can’t discover the vulnerability that the update addresses, the patch management team might need to resort to mining through the information recorded by the SMS Client hardware and software inventory agents.

The patch management team will need to determine when to release the software update and whether to allow users to influence the release process. The team might decide, for example, to allow users a seven-day grace period during which they can choose to install the update contained within the release before it becomes mandatory. The team will also have to take into account the release’s impact on the production environment. For many organizations it might be easier to release updates over the weekend instead of during the week. The team will need to factor in the size of the update, as larger updates will take longer to download to clients, especially those that connect over slow links such as portable computers on a dial-up connection.

The team might also wish to consider a staged release, where some systems receive the update before others. For example, if the update is determined to be a high priority and is applicable to the patch management infrastructure servers as well as other IT assets, the team might decide to apply the update to the servers first to guarantee their availability while deploying the update even if an exploit has been discovered.

The team will also want to draw up a timetable for the release lifecycle, including packaging the software update, testing both the update and the package, publishing the package in the patch management framework, deploying the package to systems, and validating the deployment. As part of the preparation for patch management, the organization might wish to consider developing templates of release timetables, one for each combination of category of asset and priority level. A sample timetable for a release is shown in Figure 3.

Building the Release

Once the release has been approved, the patch management team needs to build the release package, which will be distributed to affected IT assets using the patch management infrastructure. The package’s format will depend on the infrastructure tools.

Conducting Acceptance Testing

When the release package has been built, it must be tested prior to deployment to ensure that no problems will result from application of the package in the production environment. The goal of testing should be to test both the package and the update(s) contained within it in an environment that’s representative of the production environment. At a minimum, the following tests should be conducted:

• The package can be deployed successfully to assets within the production environment, including over slow links, such as those connecting remote sites and portable computers.

• The system reboots correctly after the package has been applied.

• The package can be uninstalled or rolled back.

• The package, once installed, doesn’t prevent business-critical and infrastructure systems from functioning normally.

Most organizations will wish to conduct more than these minimum tests to ensure that line-of-business applications continue to function normally. The number, range, and detail of the tests will depend on the categories of IT assets in the production environment and the software installed on them.

Before they begin to test, the patch management team members should build test plans, which define what should be tested and what the desired results should be. As with release timelines, these test plans might be developed while preparing for patch management.

Building a patch test environment that mimics the production environment, complete with patch management infrastructure, can be expensive. An alternative to a test environment is to designate certain machines in the production environment as test machines that will receive a package prior to the rest of the production environment. If the package can be deployed and the update applied successfully, the package can be deployed in a general release. This testing strategy needs to be managed carefully, as end users will become the testers and might require additional training. This strategy is also likely to result in acceptance of a patch without the rigorous testing that would be achieved in a test lab.

Leaving the Evaluate & Plan Phase and Moving to the Deploy Phase

The build of a release and the success of the acceptance testing for a package are the triggers for the change to the Deploy Phase.

4. The Deploy Phase

The last phase of the Microsoft-recommended four-phase patch management process is the Deploy phase, where the patch management team prepares to deploy the packaged software update built in the Evaluate & Plan phase into the production environment. If problems develop during, or as a result of, the deployment, the team might have to roll back or manually apply the update on affected systems. After the package has been deployed, and perhaps rolled back, the team will want to conduct a review of the experience to look for ways to improve future deployments of software updates.

Preparing the Deployment

Once the package(s) containing the software update(s) to be applied to the production environment have successfully passed acceptance testing in the Evaluate & Plan phase and the organization has moved into the Deploy phase, the patch management infrastructure and the organization need to prepare for deployment. Among the steps that need to be taken to prepare the organization are communicating the rollout schedule to affected parties and configuring the patch management infrastructure to prepare to distribute the update.

Communicating information about impending updates to stakeholders such as system administrators and end users is extremely important. A simple and effective communication tool is e-mail. By informing the organization that a software update is pending or in progress, the team can accomplish many things, including reducing the number of calls to the help desk from users wondering what’s happening to their systems and giving users the opportunity to install an update when it suits them before forcing a mandatory update on them.

The steps that the patch management team will take to prepare the patch management infrastructure for the update will depend largely on the technology used and how it’s configured. With SMS 2003, the team will need to import packages containing software updates from the test environment or build new ones in the production environment, assign distribution points for the package, and stage the updates on the distribution points. In environments where there’s a hierarchy of SMS 2003 sites, care should be taken to ensure that each package is assigned an appropriate priority and is distributed to all the sites.

Deploying the Software Update to Targeted Computers

As with the preparation, the steps taken to deploy a package containing updates will depend largely on the technology being used and the way it has been configured. The relative priority of the release and the options chosen when building the package will also figure into the deployment. Essentially, there are three steps to deploying a package containing a software update: making clients aware of the update’s availability, monitoring the package’s deployment, and recovering from failed deployments.

How the advertisement containing information about a package ready for deployment is configured will depend largely on the update’s nature. For example, a relatively low priority update might allow the end user to choose whether or not to accept the package and install the update on the user’s system, whereas a high-priority update might be mandatory and leave the user with no choice but to have the update installed. The organization might also wish to create multiple advertisements for the same package, targeting tailored advertisements at different categories of IT assets to reflect different priorities across each.

During the deployment, the patch management team needs to monitor the production environment for problems that might arise on the systems to which the package is being delivered or within the patch management infrastructure itself. Among the variables that should be tracked are the number of systems that have received the advertisement and have successfully applied the update, the number that have received the advertisement but failed to apply the update, the number that have received the advertisement but have not yet attempted to apply the update (for whatever reason), and the number that have not yet received the advertisement. Depending on the technology and how it’s configured and used, the patch management infrastructure might be able to reliably report these figures. If the infrastructure is incapable of supporting such reporting, the patch management team might have to use custom scripts (perhaps contained within the package itself) to monitor the deployment. SMS 2003 does provide a reporting mechanism so that the patch management team can gauge the deployment’s progress.

If during the deployment’s monitoring the patch management team sees a number of systems receive the advertisement but fail while applying the update, they should begin an investigation into why. Of immediate concern should be the possibility that the package is somehow corrupted. A good indicator that this is the case is a high number of failures proportional to successful applications of the package early in the deployment process. If a determination is made that the package is responsible for the failures, the team should stop the deployment and resolve the problem before making the package available again. Not all failures will be attributable to a faulty or bad package, however. Some might simply be due to the system not requiring the update contained within the package, perhaps because the system was manually updated earlier by the user or administrator or because the system was incorrectly identified as requiring the update due to a vulnerability that doesn’t exist on the system.

In most deployments there will always be a few systems that can’t be updated using an advertised package, either because of their configuration or because they aren’t managed by the patch management infrastructure. In these cases the team will likely have to visit each system and apply the update manually. Care should be taken when deploying an update in this fashion to ensure that the correct installation options are used to guarantee the updated system’s integrity.

The worst-case scenario for the patch management team is a successful deployment of a package containing an update only to find that the update needs to be rolled back due to unforeseen circumstances, such as application incompatibility that wasn’t identified during testing. Complicating factors might be that the update performs as expected on some machines but not on others. If there are no initial reports that the update causes problems to systems and the patch management process leaves the Deploy phase, the subsequent action that needs to be taken to remedy the problem can be considered a software update in its own right and requires movement through the four-phase patch management process.

Reviewing the Implementation

The goal of the review process is to gather information and lessons learned from the deployment that can be used in future applications of the patch management process. Typically held a short time after it has been determined that a package was successfully deployed, the review should focus on ensuring that the IT assets in the production environment have been categorized correctly, that the vulnerability that was mitigated by the software update has been removed, and that the update will form part of the secure baseline configuration for new systems deployed into the production environment. The review should also examine the patch management team’s performance.

Leaving the Deploy Phase

Once the Post-Implementation Review has been completed, the organization leaves the Deploy phase of the patch management process and reenters the Assess phase.