This article can also be found in the Premium Editorial Download "Storage magazine: Tips for unifying storage management."
Download it now to read this article plus other related content.
WAN throughput enhancers
The value proposition for storage mirroring and replication over TCP/IP is, of course, the reduction of ongoing operating expenses. However, TCP/IP is notoriously inefficient for storage traffic and throughput. This has led some vendors to implement performance-enhancement proxies to enhance throughput for storage replication over TCP/IP solutions. The following products all have various degrees of value to the users in helping fulfill the promise of storage replication over TCP/IP.
EMC SRDF/A for Symmetrix: EMC has enhanced SRDF with the asynchronous release. This version of SRDF disk replication software reduces bandwidth consumption by 30% because it mirrors delta sets of data every 30 seconds instead of constantly updating ordered writes (as the traditional synchronous version of SRDF does). This should alleviate some of the performance issues of the GigE director, but it doesn't go far enough. SRDF/A throughput still declines to unacceptable levels when the distance is above approximately 500 miles and especially when there are bit errors or line congestion.
Cisco Fast Step for FCIP on the MDS IP services blade: Cisco has implemented a modified version of TCP on its FCIP IP services blade in its MDS SAN switch line. Fast is designed to eliminate much of the saw- tooth effect of TCP when it encounters congestion or bit errors. Standard TCP will drop the packet size immediately back to the minimum
Cisco Fast solves this problem with two enhancements. First, it drops back more slowly to bigger packet sizes such as 512 bytes, 256 bytes and so on, and returns to the maximum speed faster. This enhancement can increase line utilization as high as 95%. These test results are on clean lines or low bit error rates as seen in the testing at Sprint. If the bit error rates are high, then the results decline significantly to an average of 50% or less. The downside is that it tends to be pricey. It costs approximately $35,000 for the blade, plus the cost of the director at around $2,500 per port.
McData fast writes for the IPS 3300, 4300 and Eclipse 1620: McData implements a fast write enhancement for iFCP. This performance enhancement speeds up mirroring operations. Test results at HDS and EMC have shown twice the IOPS as storage over IP WAN. It has limited impact on snapshots or bulk copies, and no significant line utilization gains. Distance and bit errors negatively impact throughput. For the bulk data traffic, McData implements compression. Testing from EMC and HDS showed that distance and line bit errors reduced throughput.
One interesting advantage of iFCP is its ability to limit disruptive data traffic flow between SANs. It acts like a firewall between SAN islands only allowing the desired data to move back and forth. Although this has little impact on raw data performance, it has a huge impact on the general performance of SAN fabrics themselves.
CNT's IP over UDP: CNT is the granddad of storage over IP. The company has significant experience with ESCON and FC over IP. It doesn't utilize true Transmission Control Protocol (TCP). Instead, CNT leverages its patent pending 32-bit Cyclic Redundancy Checking (CRC) on top of User Datagram Protocol (UDP) and adds compression on top of that. This gives it guaranteed delivery on the faster UDP. It also stacks SCSI commands called Command Tag Queuing. This eliminates multiple trips across the WAN link by bunching sequential SCSI commands together and sending them all at once.
Because CNT has the largest number of installations, results are available and a tad disappointing. User and some vendor testing have shown results ranging from 35% to 80% line utilization. The results tend to be better with lower bandwidth (sub DS3 speeds.) As distance increased, performance declined.
NetEx HyperIP: HyperIP is an intriguing solution because it is purely software. It is an RFC 3135 compliant TCP/IP performance enhancing proxy. Hyper IP runs on Linux OS and Intel hardware. It's a standardized TCP/IP node on the network requiring no modifications to LAN/WAN infrastructures and no proprietary hardware. The results of testing at EMC, Telstra and NTT have been nothing short of astonishing.
EMC saw SRDF achieving 90% and greater bandwidth utilization from distances of hundreds of miles (with high bit error rates on dirty lines) to as far as geosynchronous satellite distances (something SRDF was never designed to do). EMC tested throughput to as high as 450Mb/s. All of these results were without HyperIP compression turned on. Telstra's testing of HyperIP with Veritas Volume Replicator consistently saw results that doubled the throughput of the native product, regardless of distance.
HyperIP eliminates or reduces TCP/IP latency issues by terminating TCP at either end. Interestingly, the utilization test results were statistically the same with and without high bit error rates on the line. With compression turned on, testing showed line utilization rates averaging approximately 200% on DS3 to OC3 lines. The higher the capacity of the line, the less impact the compression had. OC48 lines showed no gain from the compression.
Overall, the results are impressive, but this solution is limited to storage replication solutions that run natively on TCP/IP and Ethernet. HyperIP doesn't support FCIP, iFCP or other FC SAN over WAN solutions.
Storage replication has become a popular implementation for enterprise environments because of its simplicity and speed in disaster recovery. Most of the implementations are being made over TCP/IP networks to save both capital expenditures and operating expenses, but TCP/IP is notoriously inefficient.
But one size doesn't fit all. Each enhancement has its advantages and disadvantages. The TCP/IP enhancement that seems to have the most bang for the buck appears to be NetEx's HyperIP. Of course, it's important to test any of these enhancements in your environment before buying.
This was first published in February 2004