Published: 12 Sep 2008
| Storage virtualization can reside in the fabric switch, an appliance or in the array's controller. Each architecture has its pros and cons.
Fabric-based virtualization products haven't been adopted as quickly as inline virtualization appliances like IBM Corp.'s SAN Volume Controller (SVC), but they're one of the most promising storage virtualization solutions. "Fabric-based virtualization is the best technical approach to storage virtualization but it's not taking off," says Jim DeCaires, storage product marketing manager at Fujitsu.
Switch-based virtualization brings many benefits to the SAN fabric. Because the switch-based virtualization engine is out-of-band (out of the data path), there's no need for server agents, and it's the most scalable and highest performing of all virtualization architectures.
Storage virtualization sends data to physical arrays, from single or multiple vendors, as a single storage pool with the following benefit: storage can be managed as if it was on a single array, from provisioning to advanced features like replication, snapshots and mirroring between the arrays in the pool. To accomplish this, storage virtualization products map virtual volumes to physical devices; whenever a virtualized storage resource is accessed, the virtualization layer translates and redirects storage requests to the designated physical storage according to a mapping table.
With three primary storage virtualization architectures--in-band appliances, storage controller-based and fabric-based--the location of where virtualization should occur has been hotly debated. Each approach has its pros and cons.
"The process of terminating, re-initiating and verifying I/Os adds significant latency to I/O processing," says Brian Garrett, technical director of Milford, MA-based Enterprise Strategy Group's ESG Lab. To compensate for the overhead and performance penalty of having to spawn new I/Os, products like IBM's SVC depend on cache, which adds the complexity of ensuring data integrity and data consistency in the cache, a problem switch-based virtualization products don't have.
IBM SVC is the most prominent product in this category and, through scalable cluster configurations and plenty of cache, it has greatly reduced the performance and scalability concerns that have plagued in-band virtualization products in the past. Relatively low cost, simplicity and a rich feature set have greatly contributed to in-band virtualization being the most widely deployed virtualization architecture today.
"In-band virtualization products like IBM's SVC or DataCore Software Corp.'s SANsymphony have the lowest entry cost; unlike fabric-based virtualization products, they don't require expensive intelligent switches," explains Greg Schulz, founder and senior analyst at StorageIO Group, a technology analyst and consulting firm in Stillwater, MN. Because products like IBM SVC work with any switch, in-band virtualization appliances have another advantage over fabric-based products like EMC Corp.'s Invista, which only runs on supported switch platforms.
Storage controller-based virtualization
For users who are using or have standardized on array-based virtualization, vendor lock-in is high, even more so than for fabric-based virtualization. "You wouldn't buy a Hitachi USP V for virtualization if you're an EMC or NetApp shop; but USP V virtualization would be on top of your list if you had standardized on Hitachi storage," says Schulz.
Having the array and virtualization software from a single vendor has the huge benefit of a single point of support. In stark contrast, fabric-based virtualization products, namely those from EMC and Incipient Inc., require the orchestration of three different vendors (array, switch and virtualization software vendors), which clearly carries the risk of finger pointing if problems arise.
"In a split-path virtualization architecture, 90+% of the requests pass through the switch at wire speed; only if something special like migrating of data needs to be performed [does] the control-path controller get involved," explains StorageIO Group's Schulz. The separation of the data path and control path, combined with the low latency of switching for translating and forwarding virtualized storage requests, makes fabric-based virtualization the best performing and most scalable virtualization architecture today.
On the downside, switch-based virtualization has the highest level of vendor lock-in of all virtualization approaches. Because the switch is used as the platform to run the virtualization software, it becomes very difficult for users to change switch vendors if they choose to do so. Furthermore, as intelligent switches turn into multitasking platforms, concurrent storage services from the switch vendor and third parties make supporting these switches more challenging.
As long as there are no problems, it's a great concept. But if there are problems with the virtualization software or any of the third-party storage services, the concerted effort of all involved parties may be required. Besides the relatively high cost of intelligent switches, increased complexity and more challenging technical support are among the contributing factors for the cautious adoption of fabric-based virtualization. "In general, storage managers like to keep things simple and tend to go with more self-contained, easier-to-manage solutions like LSI [Corp.'s] StoreAge or even IBM SVC," says Nelson Nahum, who was CTO at StoreAge before it was acquired by LSI.
Without question, the low latency of fabric-based virtualization is a big plus, but while it eliminates the use of cache, there is a downside: Virtualization solutions with cache, like IBM's SVC and Hitachi's USP V, use that cache to increase performance of the back-end storage. As a result, virtualization products with cache encourage the use of lower cost, lower performing storage tiers, with the cache boosting access performance. While the low latency of switch-based virtualization products is great for accessing fast arrays, its lack of cache actually turns into a disadvantage for accessing lower performance arrays. "In switch-based virtualization, back-end disk performance shows unmasked," says StorageIO's Schulz.
A second and more profound implication of the stateless nature of switch-based virtualization is the more challenging support of virtualization applications that require information beyond the mapping information. Features like remote replication and thin provisioning require memory to maintain certain state information. For instance, for a 2TB thin-provisioned volume using 100GB of physical storage, information about which 100GB are actually used needs to be maintained. While products like IBM's SVC and Hitachi's USP V maintain this information in memory along with the cache, switch-based virtualization products don't have the luxury of cache memory and their only option is maintaining this information on the SAN. "There's no complete solution for remote snapshots, remote mirroring and thin provisioning in switch-based virtualization products today because they're very difficult to implement without cache," says Fujitsu's DeCaires.
EMC Invista: Invista is the most prominent fabric-based virtualization product. While other vendors support only a single switch platform, Invista runs on Cisco Systems Inc.'s MDS, as well as Brocade switches and directors. On Cisco switches, Invista requires and runs on the Cisco MDS 9000 Storage Services Module (SSM), which provides 32 FC ports with embedded ASICs that perform the mapping and wire-speed switching of virtualized storage requests.
Invista runs on Brocade's 7600 Application Platform, available as a switch blade and a standalone appliance, with 16 FC ports with embedded ASICs. For reads and writes, the ASICs in the Cisco and Brocade modules look up the virtualization mapping information from the mapping table in memory and forward frames through the applicable FC port to the target at wire speed without the need of the CPUs on the intelligent switch module becoming involved. The control path of Invista consists of virtualization software running on the Cisco SSM or Brocade 7600, as well as the Data Path Controller (DPC) appliance. The virtualization software on the switch communicates with the DPC to receive information such as virtual disk configuration and directions for copy functions. Like all fabric-based virtualization products, Invista passes commands from the external DPC appliance to the intelligent fabric using the Fabric Application Interface Standard (FAIS) protocol.
EMC RecoverPoint: RecoverPoint is another fabric-based virtualization product that complements Invista for those customers who need remote replication or continuous data protection (CDP). In 2006, EMC acquired Kashya and subsequently released it as RecoverPoint. While Invista attempts to address a range of virtualization tasks, RecoverPoint's sole focus is on remote replication and remote site incremental snapshots via the underlying CDP engine. Invista's lack of remote replication prior to RecoverPoint is an example of the challenges fabric-based virtualization vendors face in adding features that require state information beyond the virtualization mapping table.
Fujitsu Eternus VS900: Similar to the Incipient iNSP, the Fujitsu Eternus VS900 doesn't depend on external control path appliances. An external management server is used only to upload and change configurations as well as for monitoring, but it isn't required to communicate with the virtualization software on the switch during normal operation. The Eternus VS900 also continues to operate properly even if the management server is unavailable.
The Eternus VS900 currently works only on Brocade switches. "It was developed as a collaborative effort between Brocade and Fujitsu, similar to what EMC has done with Cisco," explains Fujitsu's DeCaires. Like Invista and Incipient iNSP, the Eternus VS900 currently lacks advanced storage features like remote replication and thin provisioning.
Incipient Network Storage Platform (iNSP): Incipient iNSP is very similar to EMC's Invista with a few distinct differences. First, Incipient only supports Cisco MDS switches and directors. Data path processing is identical to that of EMC, except Incipient calls it FastPath Processor. The most significant difference to that of EMC is that all virtualization software runs within the Cisco SSM module and isn't split to have dependent code running on external appliances. This eliminates dependencies outside of the switch, making it an overall less-complex solution.
LSI StoreAge Storage Virtualization Manager (SVM): EMC, Fujitsu and Incipient virtualization products all run on intelligent FC switches. As a result, these products are expensive to deploy; bind virtualization to a switch vendor, which creates vendor lock-in; and increase the complexity of the SAN. LSI acknowledges the benefits of fabric-based virtualization, but realized early on that these disadvantages would hamper acceptance. Through two acquisitions--Storage Virtualization Manager (SVM) from StoreAge, which is the control-path virtualization software; and the LSI 8400 data-path fabric hardware from QLogic (which acquired it from Troika)--LSI can offer a virtualization solution that combines the simplicity of in-band appliances with benefits of fabric-based virtualization.
The LSI 8400 provides the data path and control path but, unlike Cisco and Brocade switches, it only provides the switching features for virtualization. This makes the LSI 8400 more cost-effective and complements existing FC switches, rather than replacing them. "The 8400 is a virtualization appliance with switching capabilities, but it's not a switch with the huge benefit that we can connect to any switch," explains LSI's Nahum.
From an implementation perspective, the LSI 8400 gets connected to an existing FC switch and the 16 switch ports become part of two zones--one contains initiator ports, while a second contains target ports. When a server accesses a virtualized volume, traffic is forwarded to the designated target port on the 8400 through a standard FC switch. The 8400 then performs the virtualization lookup and forwards frames to the appropriate storage device through one of its initiator ports. As the LSI 8400 connects through other FC switches, it adds two hops, because a standard FC switch forwards traffic to an LSI target port and then receives frames from an LSI initiator port; however, the added latency is negligible.
Provisioning and volume management
The ability to scale
There's currently no perfect virtualization product and users need to carefully weigh the product that best fits their environment's requirements. For companies that have standardized on Hitachi storage, Hitachi's array-based virtualization is likely at the top of their list. Companies with a Cisco- or Brocade-based SAN should consider Invista or Incipient; they'll also need to weigh the performance and scalability benefits against some of the challenges of these products such as their relatively high cost, complexity and feature constraints. LSI deserves consideration as it's currently the only vendor offering a virtualization product that combines the simplicity and rich feature set of an in-band appliance like IBM's SVC with the benefits of fabric-based virtualization.