Published: 03 Jun 2011
Virtual servers and storage systems don’t have to exist in separate worlds; new tools and plug-ins provide single-console management of both virtual servers and storage.
Virtualized servers have created plenty of problems for data storage managers, not the least of which is keeping track of the relationships between data storage assets and virtual servers. Some storage management products have adapted to this new environment, allowing users to keep track of virtual servers, the apps they host and the storage they’re using.
Inside every data center there are typically silos related to specific functional areas, each with a dedicated group responsible for management. There are teams responsible for managing specific data center resources, such as the network, servers, storage systems and virtualization. Each group focuses on managing its own area and works with other groups when needed to handle integration points between groups. If a new server requires shared storage, the server team works with the storage team to get storage provisioned and presented to the server.
In a traditional physical server environment, the storage group can easily manage the relationships between storage and physical servers: a logical unit number (LUN) created on a storage-area network (SAN) is assigned to a physical server and only that server uses the LUN. Server virtualization changes all that. But storage is perhaps the most critical component of a virtual infrastructure, so it must be implemented and managed properly for maximum performance and reliability. The relationship between server virtualization and storage is a tight one, so the management must be as well.
VMs can complicate storage
Virtualization is about the sharing of a common set of physical resources among many virtual machines (VMs). Virtualization file systems like VMware Inc.’s VMFS allow many physical servers to read and write concurrently to the same LUNs. This is possible because of a special locking mechanism that ensures multiple hosts have exclusive access to each of their VMs on a shared LUN. Among server virtualization’s strengths are its features that provide high availability and workload load balancing across a virtualization cluster. Features such as VMware’s vMotion and Storage vMotion can move VMs while they’re running from host to host, as well as from one storage device to another.
To further complicate things, the movement of virtual machines on storage devices doesn’t just occur at the virtualization layer. Many storage arrays now have an automated storage tiering feature built around tiers of devices with different performance characteristics such as solid-state drives (SSDs) and SATA drives that are pooled and presented to a host. The array dynamically moves data across tiers based on performance demands. All of that occurs at the storage layer and the virtual host is unaware of the movement.
While the features that move VMs around are beneficial, they can cause headaches for storage and virtualization administrators as the relationships among virtual machines, the physical hosts they’re located on and the storage device where their virtual disk resides is a dynamic one. That will have the most impact when troubleshooting problems and monitoring performance. Because the virtualization admin is unaware of what’s occurring at the storage layer and the storage admin doesn’t know what’s happening at the virtualization layer, neither one gets to see the big picture.
Plug-ins fill the management gap
Storage vendors recognized the importance of tight integration between storage and server virtualization and have worked to develop software integration with existing virtualization management tools like VMware’s vCenter Server. VMware offers a solid set of vSphere APIs that allows third-party vendors to integrate their products with vSphere. Also, vCenter Server has a plug-in architecture that makes it easy for third-party plug-ins to seamlessly integrate with the vCenter Server admin interface. Plug-ins appear as a tab inside the vSphere Client, and their behavior and appearance can be customized. This allows options or information that’s specific to a particular object like a VM, host or cluster, to be displayed.
STORAGE VENDORS' vCENTER SERVER PLUG-INS
Enlarge STORAGE VENDORS' vCENTER SERVER PLUG-INS diagram.
Not all storage vendors were quick to develop vCenter Server plug-ins, but most of the major storage vendors today offer plug-ins that allow their storage arrays to be monitored and managed from within vCenter Server. Each vendor’s storage plug-in typically only supports specific storage array models and families, and the plug-in’s functionality and features vary from vendor to vendor. Generally, storage plug-ins may offer these capabilities:
- Simplified expansion of virtual datastores. LUNs are created and presented to hosts, which then create datastores such as VMFS volumes from them. To increase the size of a datastore, the underlying LUN on the storage array needs to be increased first. The plug-in allows both the LUN and VMFS volume to be increased from the same console.
- Storage provisioning. Storage admins can assign chunks of storage to virtual environments; this allows virtualization administrators to create and size their own LUNs and manage the configuration of the storage.
- Storage management. A plug-in can give virtualization administrators the ability to manage storage array capabilities like LUN masking and thin provisioning, set multi-pathing and tiering policies, optimize I/O settings and define access lists.
- Automated VM storage mapping. This type of plug-in lets you monitor and manage the physical/virtual relationships among the virtual machines, hosts and storage arrays. This can help the virtualization admin by mapping between the virtualization identifier and the storage array identifier for the same disk.
- View detailed storage information. This brings information from the virtualization layer and the storage layer into a unified view, and lets you see the exact details of the physical storage layer from within the virtualization console.
- Physical storage health monitoring. This capability provides information on the physical health of storage arrays so virtualization admins will know when physical hardware fails or becomes degraded.
- VM cloning. The cloning of VMs is basically just a data copy job that can be offloaded to the array, which can do it more efficiently. This is especially useful in virtual desktop environments that have larger VM density.
- Backup and recovery at the storage layer. This allows you to create point-in-time snapshots on the storage array of VM datastores. You can then mount the snapshot and restore VMs from it as needed.
Anatomy of a plug-in
The joining of the storage and virtualization layers allows the virtualization admin to stay within the context of the virtual management user interface (UI) without having to grant access to a specialized storage management UI. Most of the storage plug-ins let you define credentials for the storage arrays that will be managed inside the virtualization management console. This allows seamless integration between the two consoles, and it’s also good from a security perspective as you don’t have to grant virtualization admins direct access to the storage management console.
The vStorage APIs for Array Integration (VAAI)
VMware Inc.’s vStorage APIs were introduced in vSphere to allow tighter integration of advanced storage capabilities between vSphere and third-party storage applications and devices. There are multiple categories of vStorage APIs that deal with different aspects of storage integration. The vStorage APIs for Array Integration (VAAI) were co-developed with specific storage vendors (e.g., Dell, EMC, NetApp) to enable storage array-based capabilities directly from within vSphere. Here are three examples of current VAAI features:
- Copy Offload. Virtual machine (VM) cloning or template-based deployment can be hardware accelerated by array offloads rather than file-level copy operations at the ESX/ESXi host. This can also be applied to other copy operations that occur such as Storage vMotion.
- Write-Same Offload. When provisioning an eager-zeroed virtual disk (VMDK), the formatting process sends gigabytes of zeros from the ESX/ESXi host to the array. With write-same offload, the array handles formatting the eager-zeroed thick VMDK.
- Hardware-Assisted Locking. The traditional file-locking mechanism using SCSI reservations is replaced by a more efficient mechanism that is atomic (handled in a single operation). This allows for an increase in the number of ESX/ESXi hosts deployed in a cluster with VMFS datastores.
As the vStorage APIs mature, expect to see even more offloading and features at the storage array level such as UNMAP support (return zeroed VMFS blocks), thin provision stun (stuns I/O if a thin volume runs out of space), NFS file-level copy (analogous to XCOPY on block) and NFS advanced file attributes (so vSphere can better understand more about files on NFS storage).
Hewlett-Packard (HP) Co.’s approach to the integration of storage management into vCenter Server was to leverage its Insight Control management console and integrate portions of it within vCenter Server as a plug-in. In addition to a module to manage HP storage, the company included a module to manage HP server hardware. So both server and storage hardware can be managed from a single console. When the plug-in is installed, it creates a special HP storage privilege within vCenter Server that allows access to be granted to the HP storage plug-in. The plug-in brings storage management into virtualization but not vice versa. vCenter Server has very granular permissions and roles can be defined so access to storage-specific information can be granted to storage admins. This allows storage admins to have a single console to all the storage arrays integrated with vCenter Server.
The HP Insight Control Storage Module for vCenter Server currently supports the firm’s EVA, P4000, P9000/XP and P2000/MSA storage arrays. The plug-in creates an HP Insight Software tab in vCenter Server that appears whenever a VM, host or cluster is selected; it also offers a menu option for actions such as cloning/creating virtual machines or creating datastores. The tab provides a storage overview of the object selected, such as storage provisioned to a host, and storage provisioned by a host and the arrays it’s connected to; it also provides links to directly launch the storage management console for an array. There are various views you can select to see different information such as storage disks and host bus adapters (HBAs) and paths; in addition, you can customize the columns and choose from the many detailed storage fields that are available. Sections allow you to see specific storage objects related to the object you’ve selected such as virtual machines, hosts and datastores.
In addition to information about storage arrays, there are storage tools that can perform actions such as cloning a virtual machine by utilizing array-based replication, creating batches of new virtual machines, or provisioning storage and creating VMFS volumes. While these tasks can be accomplished within vCenter Server, the HP plug-in provides automation and offloads the tasks to the storage array, which can handle it more efficiently.
The marriage of storage and virtualization in a single console allows for tighter management integration, which benefits virtualization admins, but is less beneficial to storage admins. Virtualization admins can get more involved with some of the storage-related functions, but those are traditionally handled by storage admins who may be reluctant to give up their control of provisioning and managing storage resources. Demonstrating the integration and features, and granting storage admins access to the virtualization console may help convince them to empower the virtualization admin to perform some basic storage management. Even if virtualization admins aren’t allowed to manage storage resources, being able to view detailed storage array information is advantageous by itself.
While there are plenty of management apps that integrate with VMware, there are plug-ins available for other hypervisors like EMC Corp.’s Virtual Storage Integrator for Hyper-V, which integrates with System Center Virtual Machine Manager (SCVMM). Vendors have focused on VMware because of its popularity and because VMware has a deeper and more mature set of APIs and SDKs. The storage integration plug-ins that are available for virtualization are relatively new and vendor offerings are still evolving with more features and better integration. No matter what hypervisor you have, storage plug-ins are a must-have for any virtualization environment, as they provide better visibility and integration, and enhance your ability to monitor, manage and troubleshoot your critical storage resources.
BIO: Eric Siebert is an IT industry veteran with more than 25 years of experience who now focuses on server administration and virtualization. He’s the author of VMware VI3 Implementation and Administration (Prentice Hall, 2009) and Maximum vSphere (Prentice Hall, 2010).
- Expert Tips to Address 3 Virtualization-Related Storage Issues –SearchStorage.com
- eGuide: Storage Challenges in Virtualized Server Environments –SearchStorage.com
- The Best Storage for Virtual Environments –SearchStorage.com
- Storage in a Virtual Environment: Expert Answers to 4 FAQ –SearchStorage.com