Jones Walker Law's chief information officer, Kenneth J Orgeron, believes nothing could have prepared the firm...
for the onslaught of Hurricanes Katrina, Rita and Wilma last year. However he hopes 2006 -- in which it's predicted New Orleans will again have to evacuate at least twice -- will be different.
The firm has shifted its replication site permanently to Miamisburg, Ohio, far out of the path of last year's hurricanes. It's also weighing the benefits of switching from NSI Software Inc.'s DoubleTake to XOsoft's WANSyncHA software -- which it currently uses just to replicate Exchange data.
Before the hurricanes hit last year, the firm had been replicating a little over 2 terabytes of data from some 110 Hewlett-Packard Co. 570, 530, 380 and 320 servers with DAS and internal tape libraries to an office location in Houston using NSI Software's DoubleTake. When it came time to recover Exchange data, however, there was a problem, Orgeron said..
"We had been thinking about switching before the hurricanes," he said. "We'd been doing some evaluation and testing so we already knew about XOsoft."
Between Hurricanes Katrina and Rita, Orgeron said, the firm rushed a decision in a matter of days that before had been stalling for months, switching to WANSyncHA for all its Exchange servers on the fly, while camping out in an office in Baton Rouge, La.
WANSyncHA performs automatic failover of clients from the production server to a secondary replica server, without any need to manually reconfigure clients. Once the primary server is restored, WANSyncHA allows the IT administrator to automatically failback from the replica.
Meanwhile, DoubleTake also allows replica servers to "assume the identity" of primary servers but relies on the running of scripts and command-line interfaces to make the second machine match the first one.
"For some reason, WANSyncHA seems to just work better with the Windows kernel," said Roarke Chick, manager of IT for Jones Walker, who was tasked during the hurricanes with trying to restore data with DoubleTake, as well as XOsoft's product. "It's easier to set up for Exchange, for one, and on failover it just let me click a button to tell it to go reconfigure the secondary machine -- I didn't need to run scripts or need ActiveDirectory or DNS [domain name server) experts around to try to figure out how to change the settings."
Chick pointed out that while he had liked WANSyncHA for Exchange in particular, he still found DoubleTake a superior product to the standard version of WANSync, called WANSyncDR.
Currently, the company is trying to set a window of no longer than 30 minutes downtime by June 1, the start of another hurricane season in New Orleans.
"The longest we were down during last year's season was about a day," he said. But as his story shows, that was due only to some superhuman effort and plenty of luck.
During Katrina, when the law firm's offices on the 47th through 52nd floors of an office building overlooking the SuperDome were battered by the Category 4 storm. Even high above ground in an office building, the firm was not safe from water damage. "I had water blown three feet into our offices through the blown-out windows and the walls of the building," Orgeron said.
Moreover, the city lost power, cell phone coverage and BlackBerry service soon after Katrina hit. In the end, what kept the firm's communications going was a password-protected blog Orgeron set up before fleeing New Orleans, listing contact information for everyone he could find at the firm.
"Katrina took us by surprise," he said. "What you have to remember was that it wasn't definite Katrina was going to hit New Orleans at first."
First weather predictions showed the hurricane hitting Tampa, Fla., and then over several days the path was walked over to where it would hit New Orleans. Indeed, Katrina first took out Jones Walker's Miami office before devastating its headquarters in New Orleans.
"And this all happened over the weekend. No one was at work. I made the decision to evacuate my family to Jackson, Miss., and to shut down our servers at 4 a.m. on a Sunday morning. I finally got set up with DSL in a hotel room and started trying to let everyone know where I was."
Eventually, Orgeron went back into New Orleans to recover the primary servers. He was accompanied by U.S Marshals, met at his building by the U.S. Army and wore his own sidearm for protection.
"It was almost like driving into a movie set," he said. "Tactical vehicles and soldiers and full-out combat gear patrolling the streets."
The servers were moved to another of Jones Walker's offices in Baton Rouge just in time for Hurricane Rita to hit. Rita took out New Orleans again, as well as power to the offices in Baton Rouge. By that time Orgeron was using XOsoft's WANSyncHA to replicate his Exchange data.
But by then, the firm had even bigger problems; Rita also took out power to the replication site in Houston. Jones Walker had to make hasty arrangements to shift its disaster recovery data center location to a facility run by LexisNexis in Miamisburg. One of Orgeron's employees had to load up servers from both New Orleans and Baton Rouge, both running in the Baton Rouge location following Katrina and truck them in a sport utility vehicle up to Ohio.
And they still weren't done yet. Before hurricane season was over, Wilma would do even more damage to the firm's Miami office. This time, the firm successfully recovered its data from Miamisburg using a combination of XOsoft's software for Exchange and DoubleTake for its other servers.
"What I learned is you can't always anticipate exactly what's coming," Orgeron said. "But I don't want to have to learn the hard way next time."
"Usability is something we continue to talk to our customers about relative to Exchange and SQL," said Dan Jones, vice president of North American sales and marketing for NSI Software. Jones said that from what he remembered of Jones Walker's environment, it was running a version of DoubleTake that was two years old and might not have the same features as the most current version. He also said NSI had come out with updates to its management console in the past week.
However, Chick responded that during the hurricane disaster, his engineers had downloaded the then-most-recent version of DoubleTake to try and speed the Exchange recovery.
"It was better," he said, "But not much better and certainly not better than WANSyncHA for Exchange."