Alpharetta, Ga.-based NetBank's backup and recovery systems were in need of a change, but it took a court summons landing in its lap to get the bank moving in the right direction.
The Internet-only bank, which has about $5 billion in assets and sells auto loans, mortgages and insurance, among other online services, was recently involved in a civil action with another company and had to produce particular e-mails pertaining to its relationship with that firm.
"The effort was extraordinary," said Chip Register, chief technology executive at NetBank Inc. It took the bank over two weeks to find the e-mail using all of its available resources. It had to rebuild restoration systems from three different backup applications including Netbackup from Veritas Software Corp., NetWorker from Legato (now owned by EMC Corp.) and ArcServe from Computer Associates International Inc. (CA).
The largest of its five data centers, which are dotted across the U.S., is in Columbia, S.C., and is about 4,000 square feet. NetBank manages about 40 TB of production storage on three HP Enterprise Virtual Array (EVA) systems at this location.
This data center is critical to the bank's operations, according to Register. It provides Active Director and Exchange services to over 75% of the corporation. "In a catastrophic event, we must be able to establish communication. We have offices on both coasts and the Midwest, and a lot of our workforce are out of homes. Communication is critical and e-mail is a part of that."
On top of keeping its communications up and running, NetBank falls into the category of a broker dealer, according to Securities and Exchange Commission (SEC) regulations. It must abide by Sarbanes Oxley and a host of other new laws governing data retention. "E-mail has evolved to be the system of record for activity conducted in an institution," Register said.
As soon as it was over the hill with its legal action, NetBank began the process of updating its systems. To simplify its backup procedure, it pushed out Legato and CA and went with Veritas as its sole backup supplier. Then it started researching data retention systems. "We had to be able to search for keywords in a particular timeframe, and we wanted a system that was unified in nature and presentation," Register said.
NetBank did some research on EMC's Centera, but didn't get as far as bringing it in-house. "EMC patched together a group of vendors to build that product. The presentation is not designed with the problem in mind, so it can't respond as fast," according to Register. He wasn't open to bringing startups in for such a critical system, which left HP. And NetBank was already a satisfied HP customer, or at least Register had been.
The mass exodus of HP's storage team over the past two years left Register uncertain of the company's commitment to this sector. "As a customer, I was particularly disappointed as they were strong resources … HP's hand wavered at the wheel after that, and they took the storage side of the shop too lightly. The market appropriately slapped them for that, but they have renewed their commitment," he said. "They have kept me in the know and have been very open with me, and I have confidence in them as an engineering company."
Subsequently, he bought 8 TB of HP's Reference Information Storage System (RISS), which provides 4 TB of usable capacity for capturing all the bank's e-mail. "The built-in redundancy is an inherent infrastructure cost, but it's what gives the system its incredible seek and retrieval times," Register said.
All e-mail goes through RISS first where it is captured and then sent to the corporate Exchange server. The server keeps a copy for 90 days and then expires it, but not before creating a pointer to the copy on RISS. "We had no record of e-mail that was coming in during the day but deleted before our backup at the end of that day … With RISS, users have no ability to delete their e-mail," Register said.
He pointed out that situations like the civil action NetBank just fought are not a weekly event; however, it could be common over the course of a year. Register expects to pay for the RISS system in under a year because of the reduced restoration costs. "It doesn't come cheap, but operationally, restoring data within hours, not weeks, will save us considerable expense." A top of the line, a RISS system is priced at around $400,000, but HP recently introduced a version with less capacity for $100,000.
Purchasing RISS also reduced NetBank's reliance on tape. "I am sick and tired of buying tape backup systems," Register said. The bank is on its fourth iteration of an HP tape library, which has about 800 tapes in it, and it's almost maxed out. "I'm trying to keep from buying the next one as these systems have three numbers in front of the comma, too." Because RISS stores all its e-mail, NetBank can move data that's past its retention date to tape at leisure. "Our labor costs dropped overnight," he added.
Down the line, NetBank plans to store all its Microsoft Office documents on RISS as all this stuff is up for grabs when the SEC comes knocking. "We want to be ready for them," Register said.