'The worst nightmare in any DR plan'
"Before I took this job, I was consulting for about seven years and doing a lot of disaster recovery (DR)," said Steve Perry, IT director for Costello and Sons Insurance Brokers in San Rafael, Calif. "I think the worst nightmare in any DR plan is Exchange, because it stores one large file. Once that's corrupt, you're in trouble."
"Let's say you're lucky enough that the server crashed in the morning and you get a good copy of the tape," he explained. "Then you probably wouldn't have lost any data. Building up the server again from tape will probably take you six to eight hours, provided nothing goes wrong. Then there's still some salvaging that needs to go on with rebuilding the Exchange database, and that's, you know, cross your fingers and hold your breath."
Perry evaluated NSI Software Inc.'s DoubleTake and Inmage Systems Inc.'s DR Scout products before becoming a beta tester of Mimosa Systems Inc.'s NearPoint product toward the end of April 2005.
"I've been around Exchange since version 4.0, but this technology was pretty unique," Perry said when asked why he'd replicate critical data with a fledgling startup. "If they fail as a startup company, someone's going to acquire the technology anyway."
NearPoint's software is mounted on its own server (in Perry's case, a Dell 2800 he purchased himself for the NearPoint installation, but Mimosa can OEM the hardware if necessary) and makes a shadow copy of the Exchange database on a back-end array (in Perry's case, another Dell, modified with an Adaptec controller) in continuous 5 megabyte (MB) increments.
The software then creates a "Mimosa Archive" folder in each end user's Outlook mailbox, which, when opened, connects to a Google-like search tool. End users can restore files from the archive themselves.
"Because Mimosa's writing a shadowed copy of the private Exchange store, I can just copy that over back to the Exchange server and have, in the worst case scenario, a 30 minute old copy of the server before it crashed, which is really pretty huge," Perry said.
In addition, having an e-mail archive takes some of the weight off the Exchange server itself, putting a stub file in the location of an attachment or message the user wants to access. "Our 20 users have storage needs that are somewhere around 100 times normal," Perry said. "It's not uncommon for e-mails to go out of here with 3 and 4 MB attachments. That performance boost was big for us, too."
Perry said the only drawback to Mimosa is that they've yet to add the ability to directly export .pst files, also known as Microsoft Outlook mail archives. Currently, the format is protected by Microsoft.
"You could kind of do it now, in a backhanded way," Perry said. "You'd have to import it into the Exchange server, but it's not a very direct method."
'Backing up Exchange is easy. The hard part is if you ever have to restore it'
Financial services company, NetBank Inc., handles everything from ATMs to car loans and is at the opposite end of the spectrum from Costello Insurance with two data centers in Columbia, S.C. and Alpharetta, Ga. -- but it had similar problems when it came to Exchange.
Like Perry, director of technology services Todd Warnock was already backing up his Exchange server using Veritas NetBackup but he found he needed something that made e-mail backups more serviceable.
"Backing up Exchange is easy. The hard part is if you ever have to restore it," Warnock said. "Especially if you're using tape. Even a Veritas representative said that they could make a lot of money giving away their backup product and charging an outright fee just for restores."
When the company began to consider an archive for its 10-20,000 daily e-mail messages last year, Warnock said they were courted by NAS vendors (he wouldn't say who) that took what he called "the hammer salesman approach."
He explained, "If you're trying to sell a hammer, everything looks like a nail. There were questions they just couldn't answer, holes NAS just couldn't fill."
Other vendors' products required Warnock to do some of his own assembly. Still others, he said, had unclear product roadmaps, service and support issues, or simply couldn't answer technical questions.
Warnock said NetBank finally settled on Hewlett-Packard Co.'s (HP) Reference Information Storage System (RISS) because "We thought HP's got a pretty good handle on how they do data management, and a definitive roadmap. There are places out there where there's really no good answers from anyone today -- HP's got most of the holes filled and a strategy to fill the ones that aren't."
Somewhat like NearPoint on a much grander scale, Warnock's 6 terabyte RISS system takes data from Exchange's journal mailbox and sends it to a back-end array (included, in this case, with the software from HP), where it archives and indexes the data in a separate database. It also sends data to a second RISS box in the Alpharetta data center for DR.
In addition to making Exchange backups and restores less burdensome, Warnock said, the new archive had solved performance issues, namely a 150 MB per user e-mail quota, which Warnock had trouble enforcing.
"Users would find messages deleted because they ran over the quota," he said. "Or, trying to meet the quota, they'd throw away things they thought they didn't need, only to find out later that they do need them."
The company now has a plan to implement a 180-day backup window. After 180 days, e-mail files will be moved to RISS, leaving stub files behind in users' mailboxes and freeing up the Exchange system.
"With a 180-day limit on messages, we think that'll essentially eliminate people's Exchange mailbox limits," according to Warnock.
In future releases, Warnock said he hoped HP would add support for remote and mobile users to the RISS product.
'It gave me the breathing room I needed'
Falling somewhere between the small scale of Costello Insurance and the larger financial enterprise level of NetBank is Vectron International Products, a manufacturer of oscillators. The problems for Vectron, and IT director Gene Lidman, began six months ago, when the Sarbanes-Oxley Act regulations forced a business decision -- keep everything.
It was a move that would be solid for compliance, but the company's seven geographically dispersed Exchange servers -- just four of them enterprise editions -- were soon groaning under the weight of 400 GB of storage.
Like Perry, Lidman chose NearPoint, although his implementation across multiple servers connected by a WAN is much different. The NearPoint can only process one server's data at a time, so Lidman has to queue his servers to be backed up, a process he estimates leaves him with a four hour backup window.
Still, he said he is hopeful Mimosa will add a WAN-based threading capability, and Mimosa officials said they would, though they couldn't specify a date for its release. "I'm sitting on the edge of my seat waiting for it," Lidman said. "Right now, I'm willing to wait just so I can have zero storage footprint on my Exchange servers."
A common problem
These users are far from alone -- even Microsoft itself has needed help backing up Exchange. According to the CommVault Systems Inc. Web site, Microsoft employees have been using CommVault to backup Exchange servers in its dogfood lab, where prerelease versions of Exchange Server run in production.
"Before we had any [backup] process in dogfood, whenever we had a problem, users were just out of luck," said Brian Valentine, senior vice president of Microsoft's Windows division, on a CommVault video clip. "We had to wipe their database, we had to start over, they would lose all their email -- I mean, they were just hosed."