When a new season of the regular show was about to kick off the animation staff decided to set aside the 1.2 terabytes (TB) of HD data they had produced on a 1.6 TB removable FireWire drive for safekeeping. This would also free up space on the studio's production Apple XSAN for current episodes.
After the new fall season had gotten into full swing, however, one of the show's creators came looking for the HD test footage, hoping to mull it over before discussing the concept with Comedy Central. According to J.J. Franzen, technology supervisor at the studio, that's when things fell apart.
The company went in search of help and found DriveSavers Data Recovery, a hardware recovery service provider based in Novato, Calif. DriveSavers dismantled the drive, pulled the platters out and read the data off the drive using proprietary hardware and software. (Drivesavers, like other data recovery companies, is secretive about the process of recovery, which takes place in a clean room in its facility using technology the company developed internally).
"Less than a week later, we had more than half the data back," Franzen said. "They sent the whole thing back a few days after that on a new drive."
The studio has a 15 TB Apple XSAN system, divided up into 8 TB of primary storage and 7 TB of nearline disk storage, along with 1.5 TB of what Franzen called "slop space."The company also has a full tape backup system and offsite disaster recovery (DR), but even that extensive system didn't save them in this instance.
"Our production volumes are backed up in multiple places, including a secret site offline," Franzen said. "But we figured, we'll just put this drive on the shelf for three months tops -- 'what could possibly go wrong?' "
A surprisingly popular service
Turns out it's a lot, according to DriveSavers' senior data recovery engineer, John Christopher. The company gets around 1,000 recovery jobs a month, he estimated, ranging from laptops to enterprise backup media. DriveSavers' can salvage data from tapes as well.
The company recovered data from one company in the Bahamas when a large Dell network attached storage (NAS) server and backup tapes were damaged by Hurricane Ivan in 2004; a large publishing company in the northeast also turned to the company's services when a flood in its basement ruined backup tapes.
"You'd be surprised -- 90% of our customers do backup their data," Christopher said. Sometimes, DriveSavers' services are engaged simply because of the loss of an IT employee that a company discovers is the only person that knows how to recover backup and archival data, he said.
Other times it's simple human error. "We all like to believe we can multitask," he said, but as in South Park's case, "Our brains are more focused on getting the project done -- you work hard, you work late and you don't necessarily think about backup or making sure to have multiple copies of something."
Ultimately, DriveSavers stays in business because "no method of backup is bulletproof," and sometimes, that's meant literally, according to Christopher. Another customer of DriveSavers, an employee of Oppenheimer & Co. in Manhattan brought what remained of his laptop to an office in New York after nearly being run down by an 18-wheeler on Lexington Ave.
"He tripped before he got to the curb and just managed to get out of the way," Christopher said. "But his laptop bag, which also had his cell phone and Palm Pilot inside, was crushed -- or as we like to say, the drive was compressed, but not by software."
DriveSavers competes with a very similar service from Kroll Ontrack Inc., which also offers licenses of its data recovery software for users to perform at least some of the job themselves, though most companies lack a clean room, something both companies said is often necessary for tougher hardware recoveries. Both companies were active in recovering data for businesses located along the Gulf Coast following Hurricane Katrina -- one Ontrack Data Recovery customer was interviewed by SearchStorag.come shortly after the storm hit. (See Katrina-affected business gets back Ontrack, Sept. 15, 2005).