Since the mid-90s, Ricoh has grown from four remote offices to 150 remote offices, following many acquisitions, including Lanier Worldwide Inc. and Savin Corp. Ricoh's primary backup vendor for the past four years has been CommVault Systems Inc., which it uses to protect 300 Windows clients and to back up about 16 terabytes (TB) weekly. Dave Caporaso, manager of technical services at Ricoh, had planned to roll out CommVault's Continuous Data Replicator (CDR) software to perform disk-to-disk backup at all the remote offices.
"It was nixed by the CIO, as the company is taking a step back and looking at the entire consolidation project," Caporaso said. "We're projected to go to 80-to-100 sites right now so we're figuring out how to wipe the slate clean and what's the best way to move forward."
The Savin sites Ricoh took over are running standalone Symantec Corp. Veritas BackupExec implementations to a single digital audio tape (DAT) drive. "But half of the offices just don't change the tapes … and they didn't want to admit that," Caporaso noted. "We had to put monitoring out there to uncover that." He said they found "change tape errors spewing from these sites." He added, "It's frightening, and that's where we are now."
As an interim measure, Caporaso has thrown a CommVault agent onto the most important systems at the Savin sites. "The problem is we can't back up all the servers over the wide area network (WAN), it's not doable, we had to limit it to just the home directory, with exclusions, because it was the only way we could fit the backups in, in the window we had at night. It's better than nothing," he said. Ricoh has a Hewlett-Packard Co. (HP) MSA1500 array configured to accept backups from those remote sites and then these get offloaded to tape. "It's fast, it's easy, it's a good way to go," he said.
"Life was good … Then Lanier came along and they are about the size of Ricoh and Savan combined," he said. Caporaso discovered that one of the Lanier offices had demonstration servers at remote sites that the central IT group was not supporting. "They have a Windows server that they control, not IT, which is dangerous … They save to that box instead of saving back to corporate, which gets backed up and follows an IT standard. They just do their own thing."
At the top of his list of technologies for potentially solving these backup problems is Riverbed's Steelhead wide area data services product. The Steelhead appliances would replace all tape backup systems at the remote offices and would send the data to be backed up over the WAN to the central office. Riverbed claims to be able to make WAN backup truly possible by reducing redundant data at the byte level and optimizing TCP and application protocols to make the data travel faster over the WAN, keeping backup jobs within their window.
"It's not cheap, but neither is having file servers and tape drives at every remote site," Caporaso said. "It would eliminate the need for CommVault's CDR product at all our remote sites, as we could centralize backup at corporate."
Still, his concerns about Riverbed, and wide area data services products in general, are numerous. Can the technology really perform as advertised; what management is involved and which group will end up being responsible for it? "You always hear that the management console is simple, but it never is," he said. "What's the real scoop on what it'll take to manage this product and keep it working at 100 remote sites? I guess seeing it perform with backup and our software distribution product will be the test."
Ricoh is a Cisco Systems Inc. shop on the networking side and will look at its wide area data services product as a matter of course.
"We have to move fairly quickly, as these mergers and office consolidations are going to start to happen, and as always, we don't have the luxury of time," he said.