This article can also be found in the Premium Editorial Download "Storage magazine: Overview of top tape backup options for midrange systems and networking segments."
Download it now to read this article plus other related content.
So, why now all the focus on creating interoperability labs? Some clear reasons for this trend include the increasing complexity of SAN configurations, continuing development of SAN standards and the increasing familiarity of users creating their own network configurations. As SANs have become accepted as the right way to build data center networks, there's now a greater mix of storage vendors, operating systems, switches, host bus adapters (HBAs) and other equipment that must work with each other - and as advertised.
The size of storage networks is also multiplying, leading to new, untested configurations. On top of increased complexity, there still are examples of interoperability issues between components from different vendors. Finally, users who have worked with storage network equipment have also found that they're now confident enough to mix and match their own networks, and are establishing labs to help them internally certify new applications and mixtures
The importance of testing
The benefits of interoperability testing are clear. Stephen Schaeffer, consortium manager of the University of New Hampshire's Fibre Channel (FC) and iSCSI interoperability labs, which provides interoperability testing services to vendors says, "The goal of the lab is to help vendors attain 'plug it in, turn it on' interoperability. As a computer consumer, I want something that I can unpack, turn on and use without spending days trying to configure things to connect to other things." Interoperability tests are a key to uncovering issues between different vendors, who often can't - or aren't able to - do intensive interoperability testing between their devices.
Gary Wierman, a storage analyst at Boeing, Everett, WA, who maintains a test lab with about 2TB of storage believes production is way too critical to Boeing to expose something which they haven't tested as stable. "We've found many problems in the lab we never would have anticipated without running our own tests," he says. He runs storage arrays with FC switches, testing against single hosts, with plans to test clustering environments soon.
"An interoperability lab is critical to delivering a service to a customer," says Tony Scotto, senior VP of product development at StorageNetworks, Waltham, MA, which has recently expanded its use of interoperability labs to test its new storage software. "When we were a storage service provider, we had to test the switches and storage to guarantee that the equipment would perform how a user would expect it to perform."
All of the major SAN vendors now operate their own interoperability labs. EMC, Hitachi, Compaq, and IBM all run major interoperability programs from the storage side; McData, Brocade, and others run switch interoperability labs, and even software vendors like Veritas have large investments in interoperability labs. "If you're selling storage, people need to always read back the data they wrote correctly," says Albert Cummings, who's responsible for Hitachi Data Systems' Santa Clara interoperability labs. "Customers have a very unforgiving attitude if data doesn't come back the same. It's the nature of the environment we're in that forces us to test and test and test." Vendors who expect to stay in business need to maintain interoperability testing labs, he explains, because "the alternative doesn't leave us business."
A good interoperability test is essential to assuring the performance of end-user configurations; vendors and integrators have found this is the only way they can certify a solution will work - by testing it rigorously in a lab. Most of all, interoperability labs help to ensure everything works as promised, before a large-scale rollout within an enterprise, and allows IT managers as well as vendors to make sure everything is up to snuff before committing lots of time and money into deployment of a solution.
With the large amount of vendors shipping equipment into the storage market today, there's a growing need to make sure all this disparate equipment works as required - without issues. As is the case with almost every new communications protocol, FC, iSCSI and other storage protocols have had their share of interoperability issues. Problems do occur, especially in early incarnations as protocols are being developed. Even today, with protocols like FC and hardware now into second and third generation products, there are still configurations and combinations which haven't been tried, or which vendors haven't yet certified that requires interoperability testing. Either way, interoperability tests are a requirement for developing any new network equipment, even for more mature protocols like Ethernet.
For example, one of the early issues with FC was the complexity of the Fibre Channel-Arbitrated Loop (FC-AL) protocol. The loop initialization process requires the participation and compliance of every device in the loop to operate properly. Unfortunately, the complexity of the standard and the heavy dependence on everyone getting it right often resulted in what were termed LIP storms - in reference to the barrage of low-level LIP primitives which were the symptom of this incompatibility. This problem may have single-handedly started the belief that FC had interoperability problems, and is one of the major drivers that moved the industry to switched fabric, which has none of these issues. Even as the industry has moved beyond these early gaffes, there continue to be areas where interoperability testing is required to prevent similar problems.
In fact, testing requirements have increased, not diminished. "When everything was proprietary, testing was generally very well-defined," says HDS' Cummings. "However, with open systems, we now require lots and lots of testing - so much, in fact, that it's sometimes hard to know where to stop."
As with any standardization process, SAN standards such as FC have also undergone the process of moving from pre-standard implementations of protocols to a more consistent implementation across vendors. It's arguable that this process is still under way as vendors continue to standardize such things as switch zoning and security. Fortunately for FC users, the standards and equipment have matured significantly. "As the technology matures, I expect to see a shift away from finding standards-related issues to finding implementation-related ones," says UNH's Schaeffer.
However, even as the bumps in FC have been worked out, new protocols such as iSCSI and InfiniBand will see their own interoperability issues, requiring an increasing need for interoperability testing.
The keys to interoperability testing
Interoperability testing can be done on many levels. At its most basic is testing at the protocol level, where equipment is tested to see that it meets written standards. Protocol-level testing usually involves measuring SAN equipment against the written specifications for a protocol, and ensuring they meet timing and other specifications. The industry's SANmark tests do many tests like these, and individual labs and analyzer vendors have developed test suites that check equipment against the basic assertions of a standard. This sort of testing is usually done extensively by OEMs, and rarely required of users.
This was first published in July 2002