This article can also be found in the Premium Editorial Download "Storage magazine: Exploring new options for disk-based backup."
Download it now to read this article plus other related content.
In the early days of storage networking, Fibre Channel (FC) switches with four to 32 ports served the needs of most organizations. When requirements scaled beyond 32 ports, switches were connected using inter-switch links (ISLs). This architecture gradually evolved into a so-called "core-to-edge" fabric, with larger switches in the "core" and smaller switches at the "edge" of the storage infrastructure. Data directors were reserved for "monolithic" implementations, usually focused on the mainframe.
Indeed, it's the mainframe environment from which data directors (also called channel directors) evolved. In the open-systems world, these products are called director-class switches, but the architecture is the same: More than 128 ports in a single package (frame) designed to offer high, predictable performance and at least 99.999% system availability. The core-to-edge architecture offers the advantages of incremental growth, but becomes problematic when ISLs become too numerous and latency becomes a problem. However, director-class products cost 25% to 50% more per port to implement initially because of the base cost of the frame and additional redundant hardware features. For IT organizations considering directors for the first time, or those re-examining their fabric strategy, three questions must be answered:
- Under what circumstances does it make sense to use core/edge switches, and under what circumstances should directors be used?
- Does it make sense to implement a heterogeneous fabric to get the "best of all worlds"?
- What are the key technology differentiators that should be considered between vendors?
Several key differences distinguish directors from switches. The first criterion is the number of ports. Although some 64-port devices are referred to as directors, most directors start at 128 ports. Second, directors have "non-blocking" architectures, meaning that all ports can operate at full speed simultaneously. In contrast, the internal communications of a switch are "oversubscribed" when the aggregate speed of the ports exceeds the internal bandwidth of the device. Third, directors should be capable of non-disruptive upgrades, including firmware updates. If a device must be taken offline for an upgrade, it can't deliver the 99.999% online objective of most directors. The impact of taking hundreds of ports offline for an upgrade is unacceptable to many organizations. Finally, a director should be able to isolate faults to a specific zone without impacting ports in other zones.
The competition among director-class products has never been more intense than it is now. Presently, four vendors compete in the director market: Brocade Communications Systems Inc., Cisco Systems Inc., Computer Network Technology (CNT) Corp. and McData Corp. (McData is in the process of completing its acquisition of CNT, which may occur by the time this article goes to press. CNT spokespeople weren't available for comment due to regulatory restrictions.)
Brocade. Brocade's roots are planted in the core/edge switch market it has dominated for a number of years. Its original director, the SilkWorm 12000, in many respects didn't meet the director definition (e.g., no hot code load initially and 64-port domains linked with ISLs). But the SilkWorm 24000 does, with 128 ports and non-blocking, full duplex throughput; it also uses a mesh architecture internally. Brocade boasts a unified product family from the eight-port SilkWorm 3250 edge switch up to the 128-port SilkWorm 24000.
Cisco. Similar to Brocade, Cisco groups all of its products as a unit, called the MDS 9000. The high-end director, the MDS 9509 Multilayer Director, has 1.44 Tbps non-blocking mesh throughput. Up to 224 ports can be housed in a single chassis, with up to 672 ports in a single rack. Cisco claims its chassis are designed to last 10 years, with upgrades facilitated by swapping out modules. Modules are the essence of the Cisco architecture, and may include FC, iSCSI, Fibre Channel over IP (FCIP) and multiprotocol routers.
CNT. CNT's top product is the UltraNet Multiservice Director (UMD), which features 5 Tbps non-blocking throughput with 512 ports at 4 Gbps per port. CNT refers to its architecture as a low-latency, single-stage core. Recently, however, McData, which is in the process of acquiring CNT, announced that to eliminate redundancy across the McData and CNT product lines, the UMD and FC/9000 directors will be discontinued. In addition to the UMD and FC/9000, CNT also offers the FC/FICON director with 256 1 Gbps or 2 Gbpsports, as well as the CD/9000, an ESCON channel director.
McData. McData markets its products as elements in a three-tier architecture. Its Tier 1 director, or backbone, is the Intrepid 10000, with 1Tb/sec of active non-blocking throughput for up to 256 ports and scaling to 1,024 ports. The Tier 2 Intrepid 6000, packs 140 ports in a single frame and 420 ports in a single rack. McData's directors are based on a crossbar architecture, which McData describes as providing direct, port-to-port connectivity for guaranteed throughput. Tier 3 includes McData's Eclipse and Sphereon edge switches.
Managing large SANs
SANs with more than 1,000 devices are certainly not commonplace, but they aren't unheard of. SANs with hundreds of devices are becoming common, and these large configurations introduce a variety of problems. The "any-to-any" nature of SAN architectures poses a threat to information security and increases management complexity. Of these, security is perhaps the most significant.
Most security threats in a SAN environment come from within the organization. IT groups shouldn't fall prey to a false sense of security just because storage is behind the firewall. Ports can be intentionally or inadvertently exposed to the outside, and unscrupulous admins can look for unencrypted passwords. Thus, issues such as password encryption, authentication and proper "hard" and "soft" zoning must be addressed. Each director vendor has well-developed security within its own fabrics. Brocade offers Secure Fabric OS, Cisco has the Intelligent SAN Security Suite and McData offers SANtegrity Security Suite. However, each solution is proprietary and won't interoperate with the others. The ANSI T11.3 security standard will eventually solve this problem, but it's not widely adopted; mixing different vendors' switches in the same fabric is often not advisable.
Connecting SAN islands improves management and availability (see When to move to a director, this page). While SAN islands were once the norm, most companies have consolidated, or are planning to consolidate, their SANs. This capability is enabled by functionality referred to as "Layer 3" switching. In the IP world, Layer 3 provides inter-LAN routing. Similarly, Layer 3 SAN switches facilitate routing between SANs. Obviously, security is a key component. Brocade's SilkWorm Router Module is a separate device on the fabric, as is McData's Director Service Module. Cisco's Multiprotocol Services Module plugs into the 9509 chassis, thereby reducing the number of devices in the fabric.
The flip side of linking SAN islands is dividing large SANs into logical entities that are more manageable and secure. Each vendor has its own architecture to do this: Brocade has the Logical Storage Area Network (LSAN), Cisco has Virtual SAN (VSAN) and McData has Director Flexible Partition (DPAR) (see Director differences).
Brocade LSAN. The LSAN can be thought of as a "many-to-one" approach that combines SANs using the Router Module. Thus, the SAN can be managed as one large entity or multiple smaller entities. The Router Module acts like a firewall and prevents faults from propagating between SANs.
Cisco VSAN. The VSAN can be thought of as "one to many." Specifically, one large SAN can be broken into multiples by partitioning directors, switches or fabrics. Resources can be shared across VSANs (called Inter-VSAN routing), and faults can be isolated to VSANs. VSAN technology has been adopted by ANSI as the T11.3 standard, but Cisco is the only company that supports it. VSANs can encapsulate non-Cisco products, but basically treat them as dumb devices.
McData DPAR. With DPARs, a fabric can be divided into SAN units that can be isolated from one another. McData also prescribes a process where DPARs can be used to transition from SAN islands to a unified fabric.
Virtualization and other buzzwords
For this article, virtualization is defined as "abstracting the storage infrastructure from the app so that storage becomes a 'service' to the app." As such, virtualized storage must have the intelligence to manage the location, data protection, replication and service-level delivery. As more intelligence resides in the fabric, key apps will include heterogeneous data replication, backup and recovery, storage management and data encryption. Fabric vendors don't provide these storage apps, but rather the hardware platforms they run on. Virtualization is also available from other third-party devices, usually on Microsoft Windows- or Linux-based appliances. Data encryption is also available, again using separate appliances. But the additional appliances complicate SAN deployment, use expensive ports, create additional points of failure and introduce more management software. Through the end of this decade these apps will reside on intelligent director-class switches.
An example of an application moving to the fabric is Cisco's recent announcement of its Network-Accelerated Serverless Backup (NASB), based on the Xcopy standard. With NASB, the backup data stream moves directly from disk to MDS 9XXX to tape without passing through the backup/media server. As a result, processing overhead is greatly reduced and backup speeds are increased. Only the meta data is passed to the media server for cataloging. Although NASB improves backups, it doesn't always improve restore operations. The restore data path still requires the backup/media server in most cases. The exception is volume restores using either CommVault Systems Inc.'s Galaxy or Computer Associates International Inc.'s BrightStor ARCserve. In all other cases, the restore process will see no resource reduction or speed improvement.
Competition for directors and fabric switches is driving per-port prices down at a rate of 40% to 50% annually. The market for intelligent devices is really a two-horse race: Brocade's SilkWorm Fabric Application Platform (AP) and Cisco's MDS 9000. Although McData has made some strategic acquisitions and partnerships, it doesn't currently sell an intelligent fabric device of its own. Presently, the list price for intelligent ports is around $4,000 per port, while the list price for conventional products is approximately $1,500 per port.
Cisco's intelligent switch architecture, called Network-Hosted Storage Applications, is based on a Storage Services Module that resides in the MDS 9509 frame. This module can have as many as 32 ports. The Brocade SilkWorm Fabric AP is also a blade, with up to 16 ports that reside in the 24000 director. The race is on for each vendor to attract as many OEM partners as possible. OEM qualification is a significant task and successful completion offers a substantial time-to-market advantage for the director vendor. Cisco has been qualified by IBM Corp. for its SAN Volume Controller, by Veritas Software Corp. for Storage Foundation for Networks and by EMC Corp. for Storage Router virtualization software. Brocade has also landed EMC, as well as Hewlett-Packard's VersaStor product. Eventually, both platforms will be supported by all OEM vendors.
All vendors will have 4 Gb ports available in the second half of this year. The good news is that 4 Gb technology is backward-compatible with both 2 Gb and 1 Gb technology. Thus, inclusion in the fabric will be nondisruptive. Moreover, 4 Gb modules can co-exist in director-class frames along with lower-speed modules. Of course, when mixed ports connect, the two modules will negotiate down to the speed of the slowest device.
The bigger buzz, however, surrounds 10 Gb technology. Storage managers might logically ask whether they should invest in 4 Gb now, when 10 Gb may be right around the corner. The answer is "Yes," because 10 Gb is a disruptive technology. That is, 10 Gb ports can't connect with slower ports. For the next two to three years, 10 Gb implementation will be limited to internal bandwidth to the director; indeed, some products already use it. 10 Gb may also be implemented for ISLs. Where five 2 Gb ports can be "trunked" currently to provide 10 Gb throughput between switches, a single 10 Gb ISL would obviously reduce the port consumption by 80%. But 10 Gb technology may never be implemented at the port level because non-disruptive 8 Gb technology may be available by 2007/2008 and 16 Gb technology by 2010/2011. Some highlights from product-specific roadmaps include:
Brocade. Expect expansion of the 24000 to 256 ports, the development of a Layer 3 router blade and a fabric application blade. Brocade will also continue to enhance its fabric management software.
Cisco. Although Cisco will continue to enhance the scalability and throughput of its directors and switches, expect Cisco to continue to differentiate itself by bringing network management tools to storage based on its IP networking experience.
McData. Near term, McData will be fully occupied with the integration of CNT's product line. Expect McData to continue to enhance its position in very high-performance director applications. The company will also begin to compete more vigorously as a virtualization platform in the second half of 2005.
About the author: Phil Goodwin is president of Diogenes Analytical Laboratories Inc. in Boulder, CO.