Considerations for SAN extension
By Tom Clark
Extending SAN connections over long distances enables new shared storage solutions for the open systems market, including remote tape backup, remote tape vaulting, asynchronous and synchronous disk mirroring and disaster recovery. Until recently, extending storage applications over metropolitan and wide areas meant dealing with the contradiction between the performance requirements of SANs and the bottleneck imposed by relatively slow WAN links. To attain gigabit performance, it was necessary to use dedicated dark fiber and Dense Wave Division Multiplexing (DWDM) for distances greater than six miles. Now, however, carriers are offering gigabit and multi-gigabit services for IP transport that can span thousands of miles. New IP storage products are able to leverage these services to move both Fibre Channel and iSCSI traffic far beyond the limits previously imposed by SAN extension.
Consider latency
The availability of long fat pipes removes distance and speed as barriers to SAN extension, but additional issues must be considered. Speed of light latency and latency induced by optical switches, for example, can impact the performance of upper-layer applications over very long distances. At approximately a millisecond of latency for each 100 miles, a round-trip, coast-to-coast transaction would incur about 50 milliseconds of delay. This is suitable for storage applications such as tape backup and asynchronous mirroring, but may not be tolerable for all synchronous applications. Vendors are currently testing the outer limits of such applications, but as with all technology solutions, your mileage may vary. While a coast-to-coast SAN extension is a showcase example, it is more likely that customers will deploy SAN extension within regional zones (e.g. from San Francisco to Chicago or Denver to Houston). In those instances, latencies fall to the 20-30 millisecond range with minimal impact to most upper layer storage applications.
The effect of latency over wide area SAN connections places additional requirements on the IP storage switches. Fibre Channel switches, for example, were engineered for local data center applications within a limited circumference and were not intended for long-haul operation. Given the more limited buffering of Fibre Channel switches, it is not possible to fill the pipe beyond 50-100 miles round trip. To sustain gigabit performance over extremely long distances, IP storage switches must have very deep buffers that can continue streaming frames while waiting for acknowledgements from the other end. In addition, not all applications can function in such a streaming mode. Streaming tape backup, video and content distribution are suitable, but synchronous mirroring requires testing.
Fibre Channel routing
In addition to latency issues, SAN extension solutions must accommodate the issues associated with Fibre Channel routing. As in local data center applications, connecting two Fibre Channel switches via expansion ports (E_Ports) simply extends the fabric. When connected through E_Ports, Fibre Channel switches go through a hierarchical address resolution process so that each connected switch will have its own unique address space within the single fabric. While this may function properly in a data center environment, it may pose problems when the E_Port connection is not simply from one local rack to another, but is traversing hundreds of miles. A transient disruption over the WAN link may cause a break in the E_Port connection, forcing a reconvergence of the SAN islands on each end. Each SAN island must reestablish its own addressing scheme, resulting in temporary suspension of transactions at each local site.
The creation of a single large fabric through SAN extension may also result in SAN-wide failures if disruptions occur at one of the connected sites. Since the Fibre Channel fabric is a single network segment, error conditions and state changes can propagate readily from switch to switch. This has been an issue with larger multi-switch configurations and is also a vulnerability when WAN links are introduced into the fabric.
Protocols offer solutions
While IP tunneling (FCIP) does not address this potential problem, native IP storage protocols such as Internet Fibre Channel Protocol (iFCP) and Internet SCSI (iSCSI) can leverage IP routing safeguards to both provide connectivity and prevent propagation of errors. The Open Shortest Path First (OSPF) protocol widely used in IP networks enables each connected site to be treated as an autonomous region within the network. Valid communications are allowed, but each site remains operational even if one of the sites errors or fails. For stable SAN extension, the combination of OSPF and native IP storage protocols offers the optimum solution.
About the author: Tom Clark is director of technical marketing at Nishan Systems. He is also a board member of the Storage Networking Industry Association (SNIA), co-chair of the SNIA Interoperability Committee, and author of Designing Storage Area Networks: A Practical Reference for Implementing Fibre Channel SANs. Another book, IP SANS: A Guide to iSCSI, iFCP, and FCIP Protocols for Storage Area Networks, is due out in November 2001.
