Managing and protecting all enterprise data

sommai - Fotolia

Manage storage for virtual environments

Storage management in virtual environments can be tough, but dynamic resource management and storage, network and application integration can help.

Virtual servers are flexible, mobile and can be spun up or down rapidly -- all desirable qualities that are driving the expansion of virtualization in most organizations. However, those same traits can also make storage for virtual environments difficult to manage. Hypervisor utilities or storage system monitoring tools provide a wealth of data but can overwhelm the administrator charged with maintaining performance and resource efficiency.

However, management applications designed specifically for the virtual environment can help admins sort through this data to provide intelligent analysis, proactive management and help control administrative overhead.

The problem

Server virtualization abstracts the component resources from each server -- compute, memory, storage and bandwidth -- allowing these resources to be more efficiently allocated, increasing server consolidation and potentially lowering costs. But this same abstraction creates complexity as storage administrators struggle to keep everything in balance to support larger numbers of virtual machines (VMs). And with technologies like Storage vMotion, it's like trying to hit a moving target. Like the carnival game Whack-a-Mole, administrators are trying to react fast enough to beat down problems as they pop up.

When they lack information, the tools they need or fall behind because of the sheer magnitude of the job, over-allocation is the typical result. This pushes the problem "down the road," creating a need for more resources to ensure application performance, increasing costs. But instead of reacting to problems and manually adjusting resources, management tools to handle storage for virtual environments take a better approach -- using automation and proactive analysis to prevent issues before they emerge.

Data overload

Hypervisors generate data points about a wide range of conditions in the virtual environment. These are made available to external systems, either through APIs or a software agent installed on the host. For example, hypervisors can provide details about resources used (such as CPU, host I/O and host memory) or performance metrics like queue wait times, storage response (latency) or data rates through the host's network adaptor, to name a few. These measurements are typically available for each VM and each host.

Also captured are storage metrics like data store capacity, provisioning (percentage over-subscribed), and storage IOPS, as well as data about power consumption, connection status and environmental conditions.

Hypervisor utilities can provide all this data and send alerts when VMs reach resource or performance thresholds. But that can be hard to manage, especially in environments with dozens of VMs -- and impossible when the numbers get much larger.

Management applications for the virtual environment combine these data points and display them in ways that make it easier to spot potential problems, letting admins set up dashboards with simple displays showing the status of the most critical VMs. This makes the process less reactive and enables admins to avoid out-of-resource conditions.

Dynamic resource management

These products can recommend ways to remedy problems, typically moving a VM to another host or allocating more CPU, memory or storage capacity. Some products can be set to automatically make these resource adjustments, taking the admin out of the equation and letting the system perform the balancing act. This dynamic resource tuning, whether automatic or semi-automatic, is central to most of these management products and a key value they provide. But there's more that they can do.

After addressing the immediate problems of performance and capacity tuning, companies want to take steps to prevent these problems in the future, and then work to improve overall efficiency of the virtual environment. This can lower costs, both Capex and Opex, and enable greater VM density.

Software for a virtual world

There are many options for managing virtual server environments. These include platform-specific tools, products from the storage and server vendors as well as independent solutions. Below are three examples:

  • VMTurbo is an agentless virtual appliance that connects to vCenter, Hyper-V, XenServer, or RHEV-M, scales to 10,000 virtual machines and stores two years' worth of data. VMTurbo uses the economic concept of supply and demand to describe how its software works -- workloads are ‘buyers' and resources are ‘sellers'. VMTurbo says the software is designed to maximize workload resource allocation and minimize associated cost. The company claims that this workload-centric approach allows the software to keep up with the complexity inherent in large virtual server environments. In addition to an Operations Manager, the control platform that accesses the hypervisor layer, it also has Control Modules for the storage layer, the application layer, the network layer and the container layer, enhancing the visibility of the system and improving decision-making and control.
  • Veeam ONE, a virtual server management platform from backup software vendor Veeam, connects to the virtual environment via APIs from the VMware or Hyper-V host or cluster, the vCloud Director management cell and the Veeam backup application. It can make recommendations to remediate problems or potential problems, but doesn't automate those actions. The company says that by recording and analyzing captured datasets, Veeam ONE can show trending and possible options to keep systems in balance and can model possible changes and provide a rollback. While Veeam ONE doesn't have any way to access the storage, network or other layers, it does have complete visibility into Veeam backup, providing insight about issues created by the backup infrastructure.
  • Dell Foglight for Virtualization installs as a virtual appliance supporting VMware, Hyper-V, OpenStack, and KVM, along with Citrix XenDesktop and XenApp VDI environments. Dell claims it provides the "holistic insight" required to optimize performance and allocate resources efficiently across the virtual and physical environments. It can automate all aspects of VM management, including problem remediation, plus track infrastructure changes and perform impact analyses to reduce risk before modifications are made. According to Dell, when combined with Foglight for Storage Management, administrators can get a single-pane-of-glass view of their entire virtual infrastructure, revealing issues anywhere in the storage path from VMDK to disk extent. In addition to the storage layer, Foglight provides application-layer visibility with purpose-built ‘cartridges' for all popular databases.

Management tools to handle storage for virtual environments also record data over time for analysis. This data set can identify trends in resource usage for specific virtual machines and enable better planning for system expansion or to safely load hosts with more VMs. These analyses can also help to resolve recurring out-of-resource events or aid in proactive troubleshooting.

Tools that focus solely on the hypervisor platform can be somewhat limited, as they lack real visibility into other layers in the environment -- the applications, network and external storage systems. However, some hypervisor-centric management tools are beginning to address this by integrating these other elements into the data collection process.

Storage integration

Like the hypervisor, storage systems generate data points, also available through APIs, which are used for monitoring and management. Storage resource management (SRM) tools have been using these data points for years to provide management and control in SAN and NAS environments. Virtual environment management products can use this same information to map the relationship between VMs and their underlying storage, from data stores down to storage controllers and physical disks. This information allows them to more accurately characterize the storage resources and make better decisions regarding placement, movement and sizing. The result is lower latency, fewer storage-related bottlenecks and better overall storage optimization.

Network integration

In a similar fashion, network-layer visibility can improve virtual server management decision-making as well. Virtual environments often include multiple servers running programs that depend on data from each other in real time. When workloads are positioned to accommodate compute and storage demands, these VMs can end up on different physical hosts, creating a significant amount of "east-west" network traffic. Effectively managing these situations requires understanding the relationship between these VMs as well as the network bandwidth and latency between hosts. This way, VMs can be localized to reduce network traffic and data stores efficiently placed to minimize bottlenecks.

VM sprawl and the I/O blender

Managing virtual machines is much different than physical servers. Standing up VMs takes minutes, creating a dynamic, expanding environment that's hard to keep up with. A company that may have had a few dozen servers in the past could easily have a hundred or more virtual servers to manage. The term "VM sprawl" was coined for this situation as each of these VMs represents another point of management and potential for wasted resources if not decommissioned when no longer needed.

Increased VM density can improve ROI of a virtualization project, but packing more VMs onto each physical host creates its own issues.

Fluctuating demand for performance generated by many VMs can become additive, outstripping the available resources, especially IOPS. This "I/O blender" effect can make it very difficult to balance resources, and the ability to move VMs between hosts adds another level of complexity.

Application integration

Like the network and storage layers, extending visibility up from the VM to the application layer adds valuable information about current CPU, memory and storage usage to the decision-making process. This enables VM provisioning based on the actual needs of the applications that they're running and the available supply of resources, as opposed to using best-practice thresholds. VM placement can be made based on a deeper insight and application-level resources -- such as heap sizes, database memory and thread pools -- can be configured to support quality of service goals.

Virtual server environments are getting larger and more complex all the time. By capturing and recording the abundance of data available from hypervisors and extending visibility into the storage, network and application layers, management tools to handle storage for virtual environments can provide trending analysis to help keep resources in balance as the infrastructure grows.

Article 1 of 8

Next Steps

Complete guide to storage management in virtual environments

Virtualize storage in the data center

This was last published in July 2015

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What tools have you found most effective in managing storage for virtual environments?
Are there really only three apps now that do this? I thought there were a lot more. I know there's been consolidation but that's nuts. Has anyone done any comparison testing between them?

Get More Storage

Access to all of our back issues View All