The terrorist attacks of September 11 forced all in the storage industry to rethink how they plan for disaster at individual data centers. But now that last week's blackout in the Northeast has shown that regional disasters are also a threat, it may be time to head back to the drawing board.
Last Thursday's power outage caught the majority of IT users in the northeastern United States and some parts of Canada by surprise, but according to experts, post-September 11 disaster recovery planning seems to have paid off for some users.
SearchStorage.com SAN expert Christopher Poelker -- who is also a storage architect for Hitachi Data Systems Ltd. and author of the newly released Storage Area Networks for Dummies -- said he is not aware of any problems among his user base as a result of the blackout.
"All of our enterprise clients test this kind of thing on a regular basis. We help our customers plan for this sort of thing by putting together testing procedures for data center power outages," Poelker said.
Poelker's users conduct periodic tests in which they transition from utility power to battery backup, an uninterruptible power source or building generator.
"As far as I've heard, it's all working so far. Our engineers are calling customers to see if we can be of assistance, but so far, everyone is OK," he said.
A different picture emerged from disaster recovery provider SunGard Availability Services, Wayne, Pa. The company said 62 of its customers, including "many large financial services institutions," are recovering in eight of SunGard's 38 North American locations, including New York, Philadelphia, Toronto and Chicago. More than 100 additional customers have put SunGard on alert, just in case.
The company issued a statement saying the blackout is responsible for "the largest number of declarations supported by SunGard at one time, other than following 9/11."
SearchStorage.com polled its readers and asked users affected by the power outage how much interruption in data availability they experienced. The results suggest that Poelker and Sungard are both reporting accurately on different parts of the business community.
Fifty-five percent of respondents said they had no downtime, and their disaster recovery strategies worked as they were supposed to. Eleven percent experienced minor interruptions as they switched over to backup power, and another 11% experienced disruption in the availability of minor systems. But 22% said they had major disruptions of critical systems.
Experts point to smaller organizations as the biggest problem areas. Financial institutions and large corporations have traditionally been prepared for disaster, but according to Dianne McAdam, a senior analyst for the Nashua, N.H.-based Data Mobility Group Inc., smaller IT shops have their work cut out for them.
"The events of 9/11 have caused [large companies] to possibly fine-tune their [disaster recovery] plans or more actively test their plans. The small-to-medium market is a different story," she said.
McAdam cited a recent survey from Imation Corp., in which 202 business owners, office managers and IT directors from companies with 100 or fewer employees were asked about their disaster recovery plans.
According to the survey, one in three small and medium-sized businesses is operating without a formal disaster recovery strategy in place. Sixty-four percent of companies admit that their data backup and disaster recovery plans have significant vulnerabilities.
"I firmly believe that small companies continue to remain very exposed to potential data loss," said Charlie Frey, president of QuickSAN Corp., a Rochester, N.Y., company that specializes in disaster recover for small-to-medium-sized businesses.
Frey said most medium-sized companies have good tape backup strategies in place that include off-site storage. However, small companies, those with less than five servers, continue to have a hit-or-miss strategy, one that may involve tape or CD-ROM backups.
According to Daniel Morreale, chief information officer for the North Bronx Healthcare Network (NBHN), which comprises the North Central Bronx Hospital and Jacobi Medical Center, said his systems faired very well during the blackout, but putting the disaster recovery plan to the test has helped to identify several things the NBHN can do better.
He said that additional connectivity between his two hospitals would have helped to ensure uptime.
"Our system had minimal impact and stayed up via [uninterruptible power source] and emergency generators. One file server [containing shared user files] failed with data corruption but was restored from backup," Morreale said. "Nothing tests plans like the real thing."
Shoddy disaster recovery planning can also cost smaller firms customers. Peter Cantone is vice president of strategic business development for Einsteins Mobile Upfitters, a value added reseller of commercial aftermarket products for working cargo vans and pickup trucks. He said his company's plans to launch a new Web site have been put on hold because of data loss during the blackout.
Cantone had reservations when signing with a small Web developer that had no concrete backup plans in case of a power outage. "[It] was a main concern that was never really addressed until it actually occurred. My concerns were validated," he said.
Cantone said he will be looking into a Tier 1 Web hosting provider that will include redundant backup servers that are located in different regions throughout the United States.
"The more I read about the upcoming energy crisis, the more it makes sense for us to make sure we're hosted by a large Web hosting company with multiple backup plans," he said.
Jon William Toigo, author of Disaster Recovery Planning, 2nd Edition and president of Toigo Partners International, an independent storage consulting firm, hopes that last week's blackout will improve disaster planning strategies.
"I would hope that this event underscores for people the impact of regional outages," he said. Toigo also said the event points to shortcomings of recent, fuzzy SEC rulings that he said don't require primary and secondary disaster recovery sites to be far enough away from each other. "Today illustrates that just putting them across the river in Jersey was not adequate," he said.
"It's an information economy there [in New York], and it runs on electricity."
Let us know what you think about the story. E-mail: Kevin Komiega, News Editor