Sync Up Virtual Servers and Storage

Virtual machines (VMs) can be a boon to businesses because they allow consolidation, but they can be a burden when IT considers the complexities of backing them up, and managing and tracking them. It will behoove storage administrators to learn the best ways to protect VMs in their environment and, with management and monitoring tools, control their growth.

As virtual machines proliferate, new tools help protect data and enhance performance.

As deployments of virtualized servers proliferate in enterprise IT environments, there are growing concerns associated with creating, managing and protecting these newly created virtual machines (VMs).

Because VMs are so prolific, there are problems with backing them up, managing them, migrating data among them and controlling their sprawl. But new products help IT administrators back up and protect VMs, as well as manage and monitor the connections between them and the networked storage resources they use.

Industry pundits estimate that as many as 70% to 80% of VMs rely on storage resources from Fibre Channel (FC) or IP SANs, or NAS. With such a reliance on shared storage, the problems mount for storage administrators charged with not only managing, but backing up and protecting, the environment.

Virtual machine backup blues
Chief among the concerns in backing up VMs is the cost of software licenses for backup software. Most backup vendors require a software license for each VM protected and a separate license for the physical host computer. If server virtualization is done as part of a consolidation effort, the licenses are likely already available; but if new virtual servers are being created, additional licenses may be needed.

You may also need to rejigger the backup schedule. Because the job created for backing up one physical server may encompass backing up a number of VMs, rescheduling those jobs to occur consecutively is recommended to avoid bottlenecks.

Storage admins must also tailor each backup job to the virtualization software they're using. Most backup software packages, such as EMC Corp.'s NetWorker and Symantec Corp.'s Veritas NetBackup, will back up VMware environments. There are even some, such as VizionCore Inc.'s vRanger Pro and vReplicator, that are focused specifically on VMware. These packages commonly use agent technology in which a software agent backs up each virtual and physical host machine (see "Tips for backing up virtual machines," below).

Because storage admins are most familiar with the use of agent-based backups, analysts suggest adoption of a multilayered approach to data protection, one that combines agent-based backup with consolidated or serverless backup and snapshot or cloning technology.

Another method of VM backup is the use of a proxy server. Sometimes called serverless or consolidated backup technology, the introduction of a proxy server into the environment can reduce overhead and increase performance. Backup processing is offloaded from the host computer being backed up to this consolidated backup server.

The most commonly used software of this type is VMware Inc.'s Consolidated Backup (VCB). In VCB, a series of drivers and preconfigured scripts execute the backup. A script takes a snapshot of the VM and mounts it to the proxy server. The script then quiesces the file system within the VM, while the backup software agent creates a virtual disk image of the VM. The mount is then destroyed and the virtual disk is removed from snapshot mode. Snapshot and cloning are also included in software from VMware and Citrix Systems Inc.'s XenServer Enterprise Edition (formerly XenSource XenEnterprise), as well as in many backup software packages.

Many backup vendors, including CommVault, EMC and Symantec, have recently announced the capability of their software to work with VCB. Carmine Iannace, director of IT at the Cambridge, MA-based Brattle Group Inc., has taken a combination approach to backing up his 75 virtual machines. "We install CommVault Simpana backup agents directly into the virtual guests and back them up and restore them just as if they were a physical server," says Iannace. For server failover and redundancy, Iannace uses a product from Neverfail Ltd.

"With Neverfail, if the email system malfunctions or if the site goes down in Cambridge, MA, we could resume our email functionality from Washington, DC, for the entire firm," says Iannace.

Bare-metal restoration of Iannace's VMs is also handled by CommVault Simpana. "From a physical host perspective, the VMware hosts are quite easy to recover," says Iannace. "We can easily restore those and recover the virtual guests. That's not considered a big roadblock for us."

Jim Klein, director of information services and technology at the Saugus Union School District in Saugus, CA, has also taken a combined approach. "We treat the virtual machines just as if they are physical servers with backup agents and software from the open-source software Bacula," he says.

Because images of VMs are standardized and stored on Klein's SAN, for bare-metal recovery, "we just basically recreate machines on the fly and then restore the data to it. It reduces the amount of backup tape that we need and increases our efficiency." Klein also uses a proxy server to offload processing from his host computers.

Migration of virtual machines
VMs can be deployed easily, moved around and then deleted when you're done with them. But it's that simplicity of deployment that can cause problems (see "Boot from SAN in a virtualized world," below).

VMware's VMotion enables the movement of VM images from one physical server to another for maintenance, to create new VMs and for business continuity.

Boot from SAN in a virtualized world
One of the virtues of connecting servers to shared storage is the ability to boot those servers from the Fibre Channel or IP (iSCSI) SAN. Server virtualization is proliferating, and 70% to 80% of virtual machines are attached to shared network storage that they increasingly rely on for many of their capabilities.

Boot from SAN technology evolved from the remote boot capability of Linux, Unix and NetWare NAS environments. Remote or network boots--in which the system boots over the LAN from a remote boot server--have been used for a number of years.

In the case of boot from SAN, the SAN replaces the LAN and remote boot server; via host bus adapters, the server communicates with the SAN and boots the system. When blade servers were introduced, and with them diskless blades, the need for SAN booting heightened.

Boot from SAN capability is more complex in pre-Windows Server 2003 environments. Prior to that release, Microsoft Corp. required a boot disk that was attached directly to the server with the operating system always in communication with the boot volume.

The same capability can be applied to guest operating systems running on virtualized physical servers, where it offers a variety of benefits, including the ability to consolidate servers and dispense with power-hungry disks, centralized management and speedy disaster recovery.

While there are numerous advantages to boot from SAN, there are also limitations in the Windows environment. Among them are the inability of Windows servers to share boot images--each server needs a dedicated LUN to boot and it's difficult to automate deployment of boot images across the network.

"We use VMotion not only for business-continuity purposes, but so we can do maintenance during the day," says Iannace at the Brattle Group. "We can transport the guests from one physical server to another. That has been a big plus for us since we can actually do maintenance during the day by moving the guests running on one physical host to another in real-time."

VMware recently announced that its Storage VMotion, a new feature in VMware Infrastructure 3.5, enables the nondisruptive migration of VM disk files from one data store to another.

"In VMware Infrastructure 3.5, there's also the ability to VMotion your data storage," says Ben Edson, president and CTO at VariQ Corp., Washington, DC. "Right now, data storage is still static; you can't move it anywhere unless you power down the server and do a complete move over. VMotion for Storage lets you move the actual Virtual Machine Disk Formats [VMDKs] from one storage device to another."

One of the unintended consequences of VM mobility is when a VM fails over to another physical server. Backups may fail because the schedule expects to see the original host machine rather than the new one. Management and reporting software must be adjusted to move and reschedule these mobility issues (see "VMware and N_Port ID Virtualization," below).

STORServer Inc. has unveiled STORServer Agent for VMware Consolidated Backup, software that manages backups of ESX Servers to IBM Corp. Tivoli Storage Manager servers. And Veeam Software has introduced Veeam Reporter 2.5 for VMware Infrastructure 3. The product now reports on storage capacity and utilization by VM, space allocated for each VM and disk status.

VMware and N_Port ID Virtualization
N_Port ID Virtualization (NPIV) is a Fibre Channel (FC) capability for use with virtual machines (VMs) and bladed servers that lets multiple FC initiators share a single physical host bus adapter (HBA) port.

Codified by the American National Standards Institute's Technical Committee T11 and vetted by Emulex Corp. and QLogic Corp., the NPIV standard has enabled server manufacturers such as Hewlett-Packard Co. and IBM Corp. to allow sharing of a single FC HBA among several blade servers.

With NPIV, multiple N_Port IDs are assigned to a single N_Port, making the single FC port appear as multiple virtual ports. Each virtual port is assigned its own N_Port ID and World Wide Port Name (WWPN), allowing it to have an independent route to the SAN, and enabling LUN masking and zoning for individual VMs or blades.

NPIV capability doesn't apply only to FC HBAs. Switches such as Cisco Systems Inc.'s MDS 9000 family of director-class switches or Brocade's Mi10K must also be NPIV-enabled. And the FC OSes accompanying the switches can't be left out of the loop. Cisco's IOS and Brocade's Fabric OS are NPIV-enabled. VMware Infrastructure 3 supports NPIV, as do Novell SuSE and Red Hat Linux distributions, and Microsoft Windows.

The possibilities NPIV adds to VMs are endless. With NPIV one VM may be remotely mirrored to another VM, while a corresponding VM may back up data to a remote backup server or virtual tape library. Different quality of service levels can be assigned to each WWPN. Traffic errors, packet sizes and other statistics can be isolated to individual VMs, and utilization and IO patterns predicted.

Rounding up virtual machine images
VM portability and migration also cause server sprawl. It's difficult for a storage admin to know how many VMs exist and where they're located. Because VMs are so easy to deploy, they'll spring up in the unlikeliest of places. VMs may be deployed for testing new apps, for migrating data and applications during maintenance operations, or simply to mirror data to server failover activities. They'll sprawl across the enterprise network, making it difficult to standardize their provisioning, and to maintain and patch them.

"The number of virtual machines we have at any one time varies based on development and other issues," says Iannace. "We might have times when we approach 90 virtual machines, but we are constantly building them and destroying them for either development or test. But the core 75 are what we have in production."

Another feature of VMware Infrastructure 3.5 is the Update Manager, which automates the patching of ESX Server hosts and VMs nondisruptively. Update Manager makes snapshots of VMs prior to patching; if patching isn't successful, it then rolls back to the pre-patching snapshot. The software works with Linux and Windows VMs.

"Right now, patching is a bit of a pain for ESX Server host," says Edson at VariQ. "You have to patch by individual patch release and it's fairly time-consuming. Let's say you want to patch a single physical server and reboot it; [VMware] Update Manager will in effect VMotion the virtual machines off the server, patch it and then VMotion your server image back on."

VMware is also getting into the game of managing and abating VM sprawl with the acquisition of Dunes Technologies last fall. Dunes' Virtual Service-Orchestrator 3.1 (VS-O 3.1) is server-based software that allows IT to track the creation of virtual machines and automate processes for managing the VMs.

Bus contention and performance issues
Bus contention and CPU utilization are paramount concerns in VM environments. Because all VMs in a physical server share the same PCI bus and CPUs, performance bottlenecks can crop up as VMs contend for the shared resources.

To lessen the effects of shared CPU and bus contention, it's best to implement hefty dual- or quad-core servers that are PCI Express-enabled. Adding dual- or quad-ported HBAs and Gigabit Ethernet adapters to the host servers can also help--each VM can then have its own IO channel to shared storage.

Brattle Group's Iannace deployed Dell Inc. Power-Edge servers with dual quad-core CPUs. "We've seen some high utilization of our backup servers and have upgraded them over time," says Iannace. "In fact, our main physical server is a dual-die, quad-core system--for eight cores total--because we noticed a lot of CPU utilization."

To overcome the performance limitations of virtualized servers, Klein at the Saugus Union School District says, "We generally buy a high-performance machine, such as a blade server, with extremely fast IO for a virtualization host." Klein has virtualized five Hewlett-Packard (HP) Co. blade servers with the open-source Xen hypervisor and Citrix's XenServer.

"We only use Fibre Channel for shared storage, high-speed Serial Attached SCSI [SAS] drives and fast, multicore processors," says Klein. "Since you have multiple loads hitting the same IO channel and bus, it's important that each load be able to get on and off the channel as quickly as possible."

Several vendors have introduced tools and management appliances that alleviate the performance tax virtualized servers extract from the network. Onaro Inc.'s (soon to be acquired by Network Appliance Inc.) VM Insight lets server and storage admins view the dependencies between virtualized servers and their attached storage, and determine how changes to the FC or IP network and its server connections will affect overall performance. Gear6 and Xsigo Systems Inc. offer appliances that try to mitigate virtualized server performance bottlenecks. The Xsigo I/O Director and Gear6 Cachefx create virtual IO channels between servers and storage to reduce overhead and increase IO throughput.

VMs can be a boon to businesses because they consolidate operations, but they can also be a burden when IT considers the complexities of backing up, managing and tracking them. It will behoove storage administrators to learn the best ways to protect VMs in their environment and, with management and monitoring tools, control their growth.

Dig Deeper on Storage virtualization