Real disaster recovery testing


This article can also be found in the Premium Editorial Download "Storage magazine: Lessons learned from creating and managing a scalable SAN."

Download it now to read this article plus other related content.

DR testing tools
Storage managers often ask what tools are appropriate for DR testing. Because each storage environment is so specialized--even those within the same industries--rather than dwelling on specific tools to use, storage managers should ensure that their DR testing process and procedures are stringently followed and, more importantly, improved and enhanced after each test. There are numerous tools available to emulate a user load, simulate WAN traffic and inject transmit/receive errors, and to track changes in the production environment during a DR test.

The right tool for you depends on the maturity of your testing process. For your first or second DR test, you should focus more on recoverability, including application group synchronicity, than on simulating an extreme user load or benchmarking performance. But if you're into your third year of successful testing, simulating a full production load on your servers and storage at the DR site, or even testing the resiliency of your DR fabric via simulated switch or cable failures, might be the ideal next step.

A technical aspect of DR testing that's often neglected is backup/restore. When the DR site goes live, backups of the production data that was just restored also need to commence. While the backup configuration at the DR site initially doesn't need to be as comprehensive as production, the infrastructure, software policies/procedures and schedule need to be in place

Requires Free Membership to View

right away. At the conclusion of your next DR test, pick an application, run some "transactions," perform backup operations, delete a key database and perform a restore. Backup isn't easy--you may be surprised at the difficulty of getting this done successfully, especially on the restructured architecture at the DR site.

How to make your next DR test more real
  • Insert an element of surprise--request a subtask of disaster recovery ad hoc.
  • Take an "application group" perspective to ensure it works end-to-end.
  • Switch out key team members or even entire groups because they may not be available in a real emergency.
  • Remove a key mode of communication, such as phones, and monitor the effect.

Another often overlooked element of comprehensive DR testing is documenting the steps and procedures to get back into the primary data center when it's viable to do so. If the primary data center is within the campus of corporate headquarters, there will likely be internal pressure to get back to that site if it's an option. While this may not need to be factored into the first few DR tests, be sure to consider what you would need to do to fail back to the primary site after running in the DR site for a period of time.

Eliminate the "gotchas"
If DR testing has become a mundane requisite for you and your staff, put a few more challenges into the equation, monitor the results and address what needs fixing (see "How to make your next DR test more real," at right). Plus, don't forget those other DR testing "gotchas" such as adding the same amount of storage to the DR environment when mission-critical apps acquire new production space, or forgetting that your automated DR scripts may no longer work after firmware, microcode or operating system upgrades. Make it part of your normal change control process to consider how changes might affect your DR plan.

This was first published in July 2006

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: