Corral virtual server backup

The popularity of VMware virtual servers has grown significantly in the last few years, prompting questions about how to back them up. In an exclusive excerpt from his new book, Backup & Recovery: Inexpensive Backup Solutions for Open Systems, W. Curtis Preston writes about the advantages and disadvantages of the various methods of protecting virtual server data.

This excerpt from W. Curtis Preston's new book Backup & Recovery describes the three different ways to back up a VMware server, and the pros and cons of each method.

There are several ways to back up VMware servers depending on whether the servers are running VMware Server or VMware ESX Server. In this excerpt from his new book Backup & Recovery: Inexpensive Backup Solutions for Open Systems, W. Curtis Preston writes about the advantages and disadvantages of the various methods. Preston also tells how to use bare-metal recovery to migrate to VMware, and how he turned 25 very old physical servers into "one very nice VMware server."

Backup & Recovery: Inexpensive Backup Solutions for Open Systems
by W. Curtis Preston

ISBN: 0-596-10246-1
Copyright 2007 O'Reilly Media Inc.
Used with the permission of O'Reilly Media Inc.

Available from booksellers or direct from O'Reilly Media at

The popularity of VMware virtual servers has grown significantly in the last few years, prompting questions on how to back them up. First, we'll describe the architecture of VMware and follow that with a discussion of how to back it up.

VMware architecture
VMware currently comes in two basic flavors, VMware Server and VMware ESX Server. VMware Server is a free version of VMware that offers basic virtual server capabilities and runs inside Linux or Windows. Each virtual machine is represented as a series of files in a subdirectory of a standard filesystem that you specify; the subdirectory carries the name of the virtual machine. For example, if you've chosen to store your virtual machines in /vmachines, and you have a virtual host called Windows 2000, its files will be located in /vmachines/Windows 2000.

While VMware Server runs inside standard Linux or Windows, VMware ESX Server uses a custom Linux kernel and a custom filesystem, VMFS, to store virtual machine files. You can also store virtual machine files on raw disk partitions. Neither the raw disk partitions nor files in a VMFS filesystem can be accessed by all backup commands, so you probably need to back them up in a special way.

VMware backups
When backing up a VMware machine, you have to back up the operating system of the VMware server (known as the service console on ESX systems) and the VMware application itself. You also need to back up each virtual machine's files. However, you cannot simply back up the virtual machine files or raw disks with a standard backup program. The virtual disks are constantly open and changing while the virtual machines are running and you will not receive a consistent backup of them. Even open file agents won't necessarily work properly if the virtual machines are gigabytes in size. You therefore have three options for backing up virtual machines running inside VMware:

  • Back up virtual machines as physical machines.
  • Back up virtual files while virtual machines are suspended.
  • Use VMware's built-in tools to copy a running virtual machine's files.

Back up virtual machines as physical machines
This is, of course, the easy method. Simply pretend that each virtual machine is a physical machine, and back it up as such. This method has both advantages and disadvantages.

Tip: If you use this method, don't forget to exclude the virtual machine files when you're backing up the VMware Server or ESX service console.

The first advantage of this method is that you can use the same backup system as the rest of your data center. Just because the machines are virtual doesn't mean you have to treat them as such. This simplifies your backup system. It also allows you to take advantage of full and incremental backups. Unless your backup software is able to perform subfile incremental backups, the other two methods perform full backups every day because the entire virtual machine is represented by a single file that will certainly change every day. Finally, it allows you to back up the systems live.

The disadvantage is that you have to configure backups for each virtual machine. Some may prefer to configure one backup for the entire VMware server. If you're using a commercial backup software package, this also increases your cost because you must buy a license for each virtual machine. A final disadvantage is that you also need to configure a bare-metal recovery backup for each virtual machine. Each of the other two methods won't need such a backup because restoring the virtual machine is all that's needed to perform its bare-metal recovery.

Back up suspended virtual machine files
If you can afford the downtime for each virtual machine, all you have to do is suspend a virtual machine prior to backing up its files. You can then back up the virtual machine files using your favorite backup program because the files won't be open or changing during your backup. The suspend function in VMware works just the same as on your laptop (and some servers). The current memory image and running processes are saved to a file that is then accessed when you power it on, causing all running processes to resume where they left off just before you suspended the machine.

The advantages to this method are that you don't need to configure backups for every virtual machine, and you don't need to worry about bare-metal recovery of these machines. The first disadvantage is that it performs a full backup of each virtual server every night, unless you have a backup that can perform subfile incrementals. The only open-source product that may be able to do that is rdiff-backup. The second disadvantage is that it requires suspending virtual machines, which renders them unusable during the backup. This downtime may be undesirable in many environments.

Tip: One way to have your cake and eat it too is if your virtual machines are residing on an LVM (Logical Volume Manager) volume that supports snapshots. You could suspend all the virtual machines, take an LVM snapshot, and then unsuspend the virtual machines. This minimizes the amount of time the virtual machine is suspended but allows you to take as long as you need to back up the LVM snapshot.

To suspend a running machine from the command line, run the following command, where the .vmx file is the configuration file stored in the virtual machine directory:
C:> vmware-cmd path-to-config/config.vmx suspend

You can now back up the virtual files using any method you choose. After backups have completed, you can restart the machine with the following command:
C:> vmware-cmd path-to-config/config.vmx start

Tip: If you use ESX Server, your backup tools may have trouble accessing the VMFS files. Make sure you test any new backup method.

Copy/export a running virtual machine using VMware's tools (ESX only)
Finally, if you're running VMware ESX Server, you can use the Perl APIs to copy the virtual machine files while the machine is running. The Perl APIs create a snapshot of the changes that happen to the virtual machine during the backup, storing them in a redo log, then copying or exporting the virtual files to another location. This method has the same advantages mentioned for the previous method, and it comes with the additional advantage of being able to back up systems while they're running.

This process requires using ever-changing Perl scripts, so we won't cover implementation details here. The VMware Web site includes example scripts. There's also an open-source tool called vmbk, written by Massimiliano Daneri, that can automate this process. Learn more about it at

Using bare-metal recovery to migrate to VMware

One of the really nice things about using VMware (or other virtual server solutions) is that you don't have to worry about bare-metal recovery of the virtual servers. As long as you can get them to not change during a backup, all you have to do is back up their files.

However, you can use the bare-metal recovery procedure to migrate physical machines into virtual machines. We just did this and turned 25 very old physical servers into one very nice VMware server. The following describes that migration.

I get asked all kinds of questions about backup products and how they behave on different operating systems and applications, and I use a lab to answer these questions. In addition to the usual backup hardware (SAN, tape libraries, VTLs), it consists of some Sun, IBM, and HP hardware running Solaris, AIX, and HP-UX. Until just recently, we also had about 25 Intel machines running various versions of Linux and Windows and their associated applications (Exchange, SQL Server, Oracle, etc.). I never had enough machines, and I never had the right machines connected to the right hardware. We were constantly swapping SCSI and Fibre Channel cards, as well as installing and uninstalling applications. I could have used 100 machines, but that would obviously be prohibitive in many ways. (The cooling alone would be crazy.)

So we recently decided to see if we could get rid of all these servers with VMware. We bought a white box with a 3.5GHz Dual Core AMD processor, 4GB DDR2 RAM and 1.75TB of internal SATA disks. I installed into that server two Fibre Channel cards and two SCSI cards. I then followed the alt-boot recovery method to move all of those physical servers into virtual servers, virtually upgrading each of their CPUs, storage, and memory in the process. Here are the steps I followed for each server:

  1. I used the alt-boot full image method to create an image of the entire /dev/hda hard drive to an NFS mount on the new VMware server. (These images were typically 4GB to 10GB. They were old servers!)
  2. I used VMware to create a virtual machine specifying a virtual IDE hard drive that was much bigger than the original, usually about 20GB or 40GB.
  3. I used VMware to create a virtual CD drive that pointed to an ISO file that was actually a symbolic link to an ISO image of a Knoppix CD on the hard drive.
  4. I booted the virtual machine into Knoppix using the virtual Knoppix CD.
  5. I used dd to copy the image of the real hard drive to the virtual hard drive in the virtual machine booted from the virtual CD. (We did this by mounting the NFS drive where we stored the image.)
  6. I "removed" the Knoppix CD by changing the symbolic link to point to an ISO image of a nonbootable CD and rebooted the virtual server.
  7. In almost every case, the virtual server came up without incident, and voila! I had moved a physical server into a virtual server without a hitch! One Windows server was blue-screening during the boot, but I pressed F8 and selected Last Known Good Configuration, and it booted just fine.
  8. I installed VMware tools into each virtual machine, which made their video and other drivers much happier.
  9. Once I verified the health of each machine, I changed the CD symbolic link to point to Knoppix again and booted into Knoppix. I then used either qtparted (for Linux systems), or fdisk and ntfsresize (for Windows systems) to grow the original hard drive to the new size.
With 4GB of RAM and a 3.5GHz dual-core processor, I can run about eight virtual servers at a time without swapping. I typically only need a few at a time, and what's important is that I have Exchange 2000, SQL Server X, or XYZ x.x running; they don't need to run that fast. (That's how I was able to get by with those old servers for so long.) Each virtual server can have access to either one of the Fibre Channel cards or SCSI cards, which gives them access to every physical and virtual tape drive in the lab. They will also have more CPU, disk, and RAM than they ever had in their old machine. (I can even temporarily give any of them the entire 3.5GHz processor and almost all of the 4GB of RAM if I need to, and I don't have to swap chips or open up any CPU thermal compound to do it!)

I also get to have hundreds of virtual servers and not have any logistical or cooling issues, since each server only represents 20GB–50GB of space on the hard drive. I can have a Windows 2000 server running no special apps, one running Exchange 5, one running SQL Server 7, a server running Windows 2003 with no special apps, one with Exchange 2000, and one running Windows Vista. I could have servers running every distribution of Linux, FreeBSD, and Solaris x86--and all of the applications those servers support. I think you get the point. I've got enough space for about 300 virtual server combinations like that. It boggles the mind.

Dig Deeper on Storage for virtual environments

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.