News Stay informed about the latest enterprise technology news and product updates.

Removable disks help Salvation Army recover from flood

After a pipe burst, flooding a division office data center with water and corrosive chemicals, a decision to switch from tape to disk-based backup last year proved fortunate.

Flood relief is one of the services provided by the Salvation Army. But who does the Salvation Army turn to when it becomes a flood victim? In one case last month, the charitable organization sought relief from a Unitrends Inc. disk backup system and a data recovery service.

A disaster hit the Wisconsin and Upper Michigan divisional office of the Salvation Army on March 8 and put a month's worth of data at risk. But because of a decision last year to switch from tape to removable disks for offsite backups, the office recovered 100 percent of its data from a flood that put its data center under water.

More on disk backup
Semiconductor firm turns to Diligent deduplication
Sepaton VTL customers waiting for dedupe compatibility
Don't blame the disks: Why data storage fails
Data Domain taps into Symantec's NetBackup API
IT manager Rick Thorne said a six-inch water main that fed into the basement of the building in Milwaukee suburb Wauwatosa, Wis., burst in the middle of a Saturday night. That flooded the basement where the Salvation Army kept about half a dozen servers with around 600 GB of direct-attached storage and a Data Protection Unit (DPU) from Unitrends. The water rose to a foot below the first floor of the building so quickly that flood sensors and power to the data center were cut off before alerts went out to the IT staff at home.

Thorne said the problem wasn't discovered until municipal workers received an alert from a nearby water tower that it was almost empty. "Even then, they couldn't find the burst main," he recalled. On Sunday morning, somebody spotted water pouring out of the basement windows in the Salvation Army building. The water pressure had finally burst through the windows at ground level. As if the water wasn't enough, hydraulic fluid from the building's elevator had also leaked into the basement during the flood, coating everything in corrosive chemicals.

Thorne said Unitrends and VAR Digicorp sent a new DPU right away, and the Salvation Army set up a makeshift IT headquarters in one of its community center buildings a few miles away. The organization had been backing up to the Unitrends DPU on a daily basis, and using the system to make a compressed full copy that was sent offsite every month. But the disk unit was ruined in the flood. "We immediately found the disk for January, but we couldn't find the one for February," Thorne said.

Finally it was located floating in a box underneath other debris in the basement, also coated in floodwater and hydraulic fluid. However, the 500 GB disk was eventually recovered by DriveSavers Data Recovery, a California firm that specializes in restoring data from damaged hardware. DriveSavers also recovered drives from similar situations after Hurricane Katrina.

Almost a year earlier, according to Thorne, Salvation Army replaced a DAT-4 tape autoloader with the DPU system. If they hadn't, he said, DriverSavers wouldn't even have been an option.

"Tape backups were also so slow that we were only backing up mission-critical applications," he said. "We didn't have offsite backups of all of our data on tape." Theoretically, the data could also have been restored from the production servers' internal RAID disk, but recovering a full backup image on one drive was vastly less expensive. DriveSavers charges by the amount of data restored and by the number of disks. One 500 GB disk recovery can cost close to $3,000.

Thorne said he felt fortunate that the data was recovered, and the Salvation Army is now looking to apply the lessons it learned with a new DR plan going forward. The organization hasn't made any purchasing decisions yet, but is considering more frequent offsite backups, possibly using the Unitrends Vault product for multi-site replication between its Wisconsin locations. "We've also moved the data center to the first floor," Thorne said.

Some SMBs are using software as a service (SaaS) offerings for a lower-cost, managed alternative to offsite replications schemes, but Thorne said he's not interested in that. "Working with the Vault seems like it would be more flexible -- I'd rather be able to manage and administer the system the way we want it," he said.

Dig Deeper on Primary storage devices

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.