An attentive reader recently pointed out that my last column contained an error of fact, for which new language...
has since been substituted. I want to fess up to the faux pas in this month's piece.
I had referenced a requirement then under consideration by the Security Exchange Commission and the Comptroller of the Currency that would have mandated a minimum 300-mile distance between mirrored data centers and their storage arrays. By the time the column reached print, the 300-mile rule had been tabled. When, more recently, the Federal Reserve released the "Interagency paper on sound practices to strengthen the resilience of the U.S. financial system," it had become a bad joke.
I must confess to having harbored a naive belief in the clear-headed reasoning of those who oversee our financial system. I was sure that, in these days of deifying whistleblowers and firemen, someone at SEC or OCC would feel sufficiently empowered to say those five magic words aloud: 'the emperor has no clothes.'
Having been on the ground after Sept. 11, I remember how concerned everyone was about the damage that might have been wrought to the U.S. financial system by the collateral impact of the attacks on the World Trade Center. All of the major Wall Street banks, including those in the Federal Reserve System, were forced to close their facilities -- if not because of direct building damage or the pervasive and sickening smoke emanating from ground zero, than because of milieu effects: police restrictions and telecommunications and power outages. The good news was that most of these organizations had redundant or mirrored data centers across the river in New Jersey, out of harm's way -- this time, at least.
The attacks could have been much worse. For example, in an Oct. 2003 war game conducted by Booz Allen Hamilton, the group tried to envision the consequences of a "dirty bomb" released in a port area (for details, search for "dirty bomb" at bah.com or e-mail firstname.lastname@example.org). The result was eight days of social and financial chaos within a 30-mile radius or so of the event, followed by three months of backlogged cargo deliveries. Total cost to the U.S. economy: $58 billion. And that was based on a scenario occurring in Los Angeles, rather than the financial capital of the world, New York City.
In Dec. 2001, Booze Allen played out another war game intended to discern the consequences of a release of aerosolized pneumonic plague bacteria into, of all places, Norfolk, Va. and Detroit, Mich. (See the paper here). The result was something you'd see in a disaster movie: millions of citizens isolated, dying or dead and the virtual collapse of the medical system in the face of such an unplanned burden. No cost estimates were available, only the reality that such an event would cripple a much broader area than did the Sept. 11 attacks.
You don't have to read war games to know that other disaster potentials exist that can disrupt large geographical regions. In 1989, Hurricane Hugo did what few had ever believed possible for a hurricane: the Category 5 storm blasted into the South Carolina coast then traveled inland for over 100 miles before breaking up into a week of torrential rain. Companies that had built their data centers a "safe distance" inland from such sea-borne threats found that their preparations had been for naught.
Don't forget about earthquakes. New York is in a nifty earthquake zone. In 1944, a quake in Massena, NY measured 8 on the Richter Scale and caused damage as far south as Maryland and as far north as Quebec. A magnitude 4.7 disturbance on Jan. 1, 1966, caused slight damage to chimneys and walls at Attica and Varysburg. Plaster fell at the Attica State Prison and the main smokestack was damaged (intensity 6). The total felt area was about 46,500 square kilometers.
All things considered, the idea of requiring 300 miles of separation between the primary and secondary data centers of Federal Reserve System member banks, and their mirrored storage arrays, struck me as a profoundly good one. Somewhere along the way, however, the idea fell prey to politics. Whether it was lobbyists for the banking industry, storage industry or both who stopped the initiative, the final mandate issued by the Fed was a mushy-mouthed retreat from the original zeal of the Fed's Board of Governors, OCC and SEC.
Today the document reads: "3. Maintain sufficient geographically dispersed resources to meet recovery and resumption objectives. Recovery of clearing and settlement activities within target times during a wide-scale disruption generally requires an appropriate level of geographic diversity between primary and backup sites for back-office operations and data centers. THE AGENCIES DO NOT BELIEVE IT IS NECESSARY OR APPROPRIATE TO PRESCRIBE SPECIFIC MILEAGE REQUIREMENTS FOR GEOGRAPHICALLY DISPERSED BACK-UP SITES (emphasis added). It is important for firms to retain flexibility in considering various approaches to establishing backup arrangements that could be effective given a firm's particular risk profile. However, long-standing principles of business continuity planning suggest that backup arrangements should be as far away from the primary site as necessary to avoid being subject to the same set of risks as the primary location. Backup sites should not rely on the same infrastructure components (e.g., transportation, telecommunications, water supply and electric power) used by the primary site. Moreover, the operation of such sites should not be impaired by a wide-scale evacuation at or the inaccessibility of staff that service the primary site. The effectiveness of backup arrangements in recovering from a wide-scale disruption should be confirmed through testing."
In short, after several months of deliberation, the Fed reached the same conclusion that was a given at the outset of deliberations: mirrors in close proximity were subject to the same geographic threat. However, not only did the Fed stop short of placing a stake in the ground with respect to recommended site dispersal, it abdicated from the role of making any prescription whatsoever now or in the future. All in all, the document, intended to safeguard the U.S. banking system from catastrophic interruption, was less a mandate than a collection of motherhood considerations.
Bottom line: I erroneously reported that a 300-mile mirror pair separation rule was in the offing. It should have been, but it isn't.
So those of you who are content with your synchronous mirrors within 30 miles of their primary arrays can forget about confronting a pesky old dispersion mandate any time soon. The Fed isn't going to come to your door with a warrant. You can rest easy now...or can you?
About the author: Jon William Toigo has authored hundreds of articles on storage and technology along with his monthly SearchStorage.com "Toigo's Take on Storage" expert column and backup/recovery feature. He is also a frequent site contributor on the subjects of storage management, disaster recovery and enterprise storage. Toigo has authored a number of storage books, including "Disaster recovery planning: Preparing for the unthinkable, 3/e".