Published: 26 Sep 2008
| There are three main ways to back up virtual servers. Here's how to determine which method is best for your storage requirement.
SERVER VIRTUALIZATION IS supposed to simplify IT, and it does, except for one little caveat: It currently complicates backup and recovery. Companies quickly discover they can back up their virtual servers the same way they do their physical servers, but they may not get the same results.
"It's simple to back up virtual servers if you treat them like physical servers," says Scott Polly, director of technical publications at Vizioncore Inc., Buffalo Grove, IL. The firm provides vRanger Pro, a GUI-based application that automates much of the command line scripting typically involved in VMware backup. Simple, but "you don't get the benefits of server virtualization," adds Polly.
The benefits Vizioncore's Polly refers to revolve around efficient management. If you treat virtualized servers like physical servers, you have to manage each individually, which undercuts any improvements in management efficiency. You also have to buy and install a backup agent on each virtual server as if it were a physical server, which will certainly require more work and may entail additional license fees depending on your backup product. In short, storage admins face more than the usual backup complications because there are more virtual servers and they have to run multiple concurrent backups.
In addition, the immaturity of server virtualization makes the backup challenge more difficult. VMware is still feeling its way when it comes to backup and recovery. "It's hard to get clear documentation," reports John Dolan, principal consultant at Viant Solutions in Suwanee, GA. Dolan had been trying to pin down the backup of 24 virtualized servers spread across three hosts at Perimeter Church, a mega-church based in Duluth, GA, but getting accurate information was frustrating. Early on, he feared he might need to write some software to make the church's version of Symantec Corp.'s Backup Exec (Version 10d) work with VMware ESX 3.0.2.
The solution turned out to be simple once he had the correct information. "Backup Exec [Version 10d] simply isn't supported by ESX 3.0.2," says Dolan. He finally discovered that after digging through VMware's latest compatibility guide. "We needed to be running Backup Exec 11d or later," he says. That meant Perimeter Church had to upgrade to the next release, which entailed an added cost. "But we'll only need one server license plus the Exchange and SQL license for our virtual servers," he adds.
The VMware backup picture is changing almost weekly as VMware pushes out new documentation and products, as more vendors scramble to certify their backup tools with VMware and as consultants begin to identify some best practices (see "Virtual backup best practices," below). VMware insists that backup isn't difficult: "It's not hard, but different," says Jon Bock, VMware's senior product marketing manager, adding that "it will change what you're probably doing now."
Except that backing up one server is different than backing up five, 10 or 15 virtual machines (VMs). The problem is resource contention. "Backup imposes overhead on the [physical] server," says David Dale, chair of the SNIA Canada IP Storage Forum, a member of the SNIA board of directors and a NetApp executive.
Regardless of how many virtual servers are backed up, they're guests of a single physical server and must share the CPU and network resources. One possible solution, suggests Dale, is to "delegate the backup overhead to the array."
VMware's solution is different: VMware Consolidated Backup (VCB). "VCB is a Windows-based proxy host," says Scott Miller, president at Server Centric Consulting in St. Louis. Instead of installing individual backup agents on each VM, you do the backup on VCB, which offloads the backup process overhead to the physical proxy host, which is a dedicated server.
Some observers think the resource contention issue is a red herring. "In theory, it's possible that you could overload the physical machine by doing multiple backups, but we haven't hit that particular pain [point] in the real world," notes Ashley D'Costa, enterprise solutions architect at Mainland Information Systems, Calgary, Alberta, a Canadian systems integration and consulting firm. The virtualization backup problems he encounters have more to do with performance and data consistency.
W.R. Berkley Corp., based in Greenwich, CT, backs up almost 100 VMs running as LPARs on its IBM Corp. pSeries servers with Symantec's Veritas NetBackup agent installed on each VM. "We haven't hit any resource contention yet. I guess we could, but the boxes have a ton of ports," says Tom Whelans, VP of operations at the property and casualty insurer. The backup software is licensed by physical server so it doesn't incur additional licensing charges.
Also challenging are the different types of recovery: backing up and recovering the entire VM as a file, or backing up and recovering single files. To back up and recover the entire VM, you "copy it block by block, disk to disk and capture the VM in a specific state," says Oglesby. "Here, you have to restore the entire VM as a whole."
To back up and recover individual files you "do a file-level backup within the VM using a normal agent or VCB, like a traditional backup. In this case, you can restore a single file easily but can't bring the entire VM server back online easily," he continues. You have to recover the VM piece by piece.
Virtualization is essentially file-oriented; the VM consists of one or two encapsulated files. Restoring individual files remains a challenge. It's easy to back up and restore the entire VM. Restoring individual files isn't straightforward and, depending on the backup tools involved, may be quite cumbersome and involve restoring the entire VMware Disk file (VMDK).
"Everybody wants to rapidly back up at both the granular [file] and virtual machine level," says Mainland Information Systems' D'Costa. With VMware as it's configured today, that's not possible. "Right now you have to choose one or the other," he says. Granular backup means you can recover individual files without restoring and mounting the entire virtual server. With machine-level backup, you recover the entire virtual server as a single VMware encapsulated file; VMDK is fast and easy if you need to recover the whole thing.
VM backup strategies
D'Costa identifies three main approaches to backing up virtualized servers: conventional server backup with the use of individual backup agents on each VM; use of the backup management tools provided by the virtualization vendor; and the use of a standalone backup proxy server, such as VCB, or appliances from third-party vendors.
A storage manager might be tempted to simply back up the VMs on the physical host like regular files. In this case, the organization would back up each VM's VMDK, a large disk file containing the VM configuration and data. Nice idea, but it might not work well. Unless the VM is shut down, you might back up data in use, which is likely to result in inconsistent data.
The use of an individual backup agent on each VM can avoid data inconsistency by quiescing the app during backup. However, this approach can result in high backup software licensing fees if the organization must purchase an instance of the license for each VM. It also creates a potential for resource contention unless the organization staggers its virtual backups. Still, the advantages of putting a backup agent on each VM are simplicity and familiarity. The backup procedure runs no differently and admins can do all of the usual things, such as file-level recovery, and full or incremental backups.
However, the backup application, unaware of the encapsulated nature of the VMDK files, can't do things it otherwise might do to speed backup, notes Mainland Information Systems' D'Costa. This approach also undermines some of the efficiencies organizations hoped for from server virtualization in the first place as each agent must be managed individually.
D'Costa's second approach runs the backup software in the virtualization server itself, such as VMware ESX. This will likely be a Linux backup agent capable of backing up all of the VMs. However, this results in image-level backups that, although fast, don't allow for the easy recovery of individual files. It also requires scripting to automate the shutdown, snapshot and restart of the VMs.
The proxy server (particularly VCB), D'Costa's third approach, may become the most popular. The proxy server moves the backup from agents on the VMs or from the virtualization server to a dedicated server. In the case of VMware, "you put the backup agent on VCB and back up all the VMs," says Server Centric Consulting's Miller.
The proxy server is typically a Windows server connected to the same SAN volumes as VMware's ESX server. "With VCB you can now back up multiple VMs on the same physical host and VCB organizes the backups to keep them from overusing the resources," adds Miller. VMware provides a load balancer called Distributed Resource Scheduler (DRS).
But VCB is far from a backup panacea. D'Costa points out that it requires a number of pieces to work right: a sync driver on the ESX server to flush the file systems and create a snapshot, a vLUN driver on the proxy server to present VMDKs to the proxy, and command line utilities to assist with scripting automation through the command line interface (CLI). It also will require an integration module provided by VMware or the backup application vendor.
Storage managers often mix different approaches. For example, Rockledge, FL-based Health First is a three-hospital healthcare provider network serving Brevard County. It uses VCB through a third-party tool, Vizioncore's vRanger Pro, to back up 1.5TB of data from approximately 220 VMs every night. The data is backed up to disk. However, the hospital also uses IBM Tivoli Storage Manager (TSM) and installs a TSM backup agent on some of the VMs. The TSM backups go to disk and then to tape, which is shipped offsite.
VCB and vRanger Pro let the team "deal with the growth of VM sprawl," says Joel Otero, network engineer at Health First. Some days "we're building 10 to 15 VMs. Tivoli couldn't keep up with that. It's far too much to script," he explains.
W.R. Berkley combines VCB with Veritas NetBackup and FalconStor Software Inc.'s Network Storage Server for backup and recovery of its virtualized Windows and pSeries servers. The FalconStor product takes snapshots a few times a day and replicates them to a second data center. "So we risk, at worst, losing a couple of hours of data," says W.R. Berkley's Whelans. Using VCB, the company can restore individual files. FalconStor also spins off the snapshots to tape every night.
Virtualization backup tools
VMware recently introduced Site Recovery Manager to automate disaster recovery management.
Site Recovery Manager works with VMware's Virtual- Center management console and with replication software from various storage partners, including 3PAR Inc., Dell Inc., EMC Corp., FalconStor, Hewlett-Packard Co., Hitachi Data Systems, IBM, LeftHand Networks Inc. and NetApp.
Navicure Inc., a Duluth, GA, provider of revenue-cycle management systems for physicians, relied on extensive manual scripting to back up as many as two dozen virtual servers using the replication capabilities built into its EqualLogic SAN (now owned by Dell). "The scripting was taking us hours," says Donald Wilkins, Navicure's IT director. "We wanted to streamline the process." Just trying to recover a VM and promote it to production involved a cumbersome process of mounting files and changing IP addresses.
"Site Recovery Manager automated all this for us, all the scripting and changing of IP addresses. It also lets us test our DR plan non-intrusively," says Wilkins.
To validate this approach, Wilkins undertook the two-hour challenge--to bring up 10 VMs in two hours. "We started at 7pm on a Friday night and began cloning VMs with EqualLogic. We changed IP addresses, replicated them and defined protection groups in less than two hours. We pressed a button and had all 10 VMs up and running 10 minutes later," he says. By 9pm the team was heading home.
UGL Unicco, Newton, MA, turned to STORServer Inc.'s STORServer Appliance for VMware Consolidated Backup. With approximately 100 VMs running on five VMware ESX servers and TSM as its primary backup tool, UGL Unicco uses the STORServer Appliance to interface with Tivoli, VMware and VCB. For most of its VMs, it puts a TSM agent on the server and backs up in the conventional way to STORServer disk, which spins it off to LTO-4 tape. It uses VCB with approximately 30 critical VMs to allow for file-level recovery.
"We're an IBM shop and we liked what Tivoli does, but it's very complex, highly scripted and uses a CLI, so we went with STORServer as a GUI front end," says Darrell Stymiest, UGL Unicco's network services manager. VMware backup was similarly challenging, requiring extensive scripting through another CLI. With the STORServer Appliance for VMware Consolidated Backup "we can take the VCB snapshot and write it to disk on STORServer and then send it to tape," says Stymiest. Recovering files with VCB remains a cumbersome multistep process, but it's much improved over previous VMware backups. "We want to use VCB with 90% of our VMs eventually," he adds.
"Backing up virtual servers isn't simple and the tools aren't perfect," says Mainland Information Systems' D'Costa. But the virtualization backup process is still immature. "Backup tool vendors and array vendors are still trying to conform with VMware," which is still evolving its tools and APIs, adds D'Costa. Until the virtualization industry matures in a few more years, backing up VMs will remain a challenge.