Remote DR: faster, farther and cheaper

Post-Sept. 11, you need to consider disaster sites that are geographically distant from your main data centers. Remote replication software, IP storage and new techniques for long-term storage are changing the DR distance equation.

This article can also be found in the Premium Editorial Download: Storage magazine: Tips for unifying storage management:

Storage-to-storage WAN throughput enhancers working with EMC's SRDF

Cisco MDS and McData products are switches that connect to the EMC Symmetrix via Fibre Channel (FC) and to the IP network via GigE into an Ethernet switch.

The CNT's UltraNet Edge Storage Router (UESR) is a gateway that connects to an FC switch, which is connected to the EMC Symmetrix and to the IP network via GigE or FAST Ethernet to an Ethernet switch.

The NetEx HyperIP software solution is an RFC 3135 TCP/IP performance enhancing proxy with no FC connection. It connects to the IP network via GigE, then onto a GigE director and finally to the EMC Symmetrix. Basically, it is Symmetrix to HyperIP over WAN to HyperIP to Symmetrix.

Sept. 11, the 2003 blackout and daily computer viruses are all grim reminders that every company needs a business continuity plan.

Such plans are analogous to personal disability insurance. It costs a ton of money and no one wants to use it, but if misfortune strikes, incoming revenue is protected. The key is minimizing the insurance costs against potential losses in a disaster.

Increasing regulatory demands and reducing costs tend to be diametrically opposed objectives. Fortunately, there are five trends that have converged that allow IT organizations to meet regulatory requirements while reducing costs. They are:

  • New disk mirroring and replication choices over distance
  • Low-cost secondary disk storage with the widespread adoption of less-expensive Parallel ATA (PATA) and Serial ATA (SATA) drives in the enterprise
  • Declining costs of bandwidth
  • The use of IP networks for business continuity
  • Storage-to-storage over WAN throughput enhancers
Replication choices
During the last decade, EMC Corp.'s Symmetrix Remote Data Facility (SRDF) set the bar for regulation-compliant storage-to-storage disk mirroring and replication solutions for business continuity and disaster recovery. It created a paradigm shift in business continuity that was far ahead of the competition. Before SRDF, most enterprise operations backed up to tape. If the IT organization was truly advanced, they backed up electronically to an automated tape library across the WAN.

SRDF was one of the few options that allowed transactions to be mirrored synchronously, semi-synchronously and asynchronously locally and across a WAN to a remote facility. To perform remote mirroring, it required ESCON channel extenders from either CNT or Inrange. SRDF now works over Fibre Channel (FC), and the FC to WAN gateways are from Ciena, McData Corp. and others for remote sites. SRDF also works natively on TCP/IP and Gigabit Enternet (GigE) with EMC's new GigE Director for EMC's Symmetrix arrays.

Eventually, all advantages come to an end, and SRDF is no exception. True Copy from Hitachi Data Systems (HDS), CASA and DRM from Hewlett-Packard (HP), PPRC from IBM Corp. and other products have matched, and in some cases, exceeded the capabilities of SRDF. Third-party software from DataCore, FalconStor, NSI, StoreAge, Veritas Software Corp., Zetta and others also performs similar functions. The new products brought not only more choice, but competition and lower prices.

Nearline storage
The introduction of nearline storage utilizing lower cost IDE drives (PATA and SATA) has made storage replication significantly less expensive. These drives are considered less robust and reliable than SCSI or FC drives. They also typically run slower at 5,400 RPM (PATA) and 7,200 to 10,000 RPM (SATA). This makes them ideal for backup storage. These devices don't have to withstand the rigor of day-to-day use. In the event of a disaster, they will be deployed to bring the organization back to a serviceable status as quickly as possible. Even if that service is in a somewhat degraded state, it will be in an acceptable state.

Declining bandwidth costs
The cost of WAN connections has declined over the past few years. In some cases, the declines have been as much as 75% over the last two years. Expectations are that bandwidth costs will continue to decline (albeit at a slower rate) because the supply continues to exceed demand.

In spite of this ongoing trend, large bandwidth, DS3, OC3, OC12, OC48 and OC192 are still expensive. Storage replication requires large amounts of bandwidth. Shared or dedicated bandwidth makes no difference. Just calculating the bandwidth makes this clear. For example, replicating 1TB of storage would take nearly 50 hours over a DS3, assuming 100% bandwidth utilization. Using the same assumption, it would take approximately 14.4 hours for an OC3 and 3.6 hours for an OC12. In the real world, 100% bandwidth utilization isn't likely. The reason is TCP/IP, which is the preferred storage-to-storage WAN protocol of choice. There are three reasons for this:

  • There's a perception that the bandwidth is free because the TCP/IP WAN links already exist for interactive traffic. The conventional rationale is that storage-to-storage replication, or snapshot, typically occurs at night or on weekends when the majority of users are not utilizing the network. This allows the already existing TCP/IP WAN links to be leveraged by the storage applications without negatively affecting other applications.
  • A dedicated, separate storage replication WAN isn't required. No separate WAN means no separate management.
  • Any additional bandwidth implemented for the storage replication can be shared by the interactive TCP/IP applications.
Unfortunately, there's no free lunch. One hundred percent bandwidth utilization isn't very likely. TCP/IP rarely exceeds 30% bandwidth utilization and in congested or high bit error rate environments, is often less than 10%. This means the amount of bandwidth must be increased by three to 10 times to meet the same throughput.

Storage over IP
One of the early applications for storage over IP is mirroring. The three IP standards that have emerged are Fibre Channel over IP (FCIP) Internet Fibre Channel Protocol (iFCP) and iSCSI.

FCIP is an Internet Engineering Task Force (IETF) standard that encapsulates FC frames in TCP/IP packets and places them in GigE frames. It's used to connect FC SANs over WANs for data replication.

The iFCP standard assigns a TCP/IP address for each FC frame and also packages them typically in GigE frames; currently only McData offers this. The key advantage for iFCP over FCIP is the ability to selectively move data from specific devices between the SANs.

In other words, a Register State Change Notification (RSCN) fabric disruption on one SAN disrupts the fabric on the connected SAN. The RSCN broadcast is the FC switch methodology for informing all devices on the fabric that a change has occurred. Larger fabrics contain more RSCNs, which disrupt data flows. Isolating interconnect SAN RSCNs from one another reduces the impact on the fabric and improves performance.

Fabric Shortest Path First (FSPF) is the FC fabric standard routing algorithm. The FSPF database size is correlated to the number of devices, paths, switches and ISLs in a SAN fabric. When FCIP interconnects two or more SANs into one, this increases the FSPF database for all switches, slowing performance. When iFCP interconnects two or more SANs, it only passes the address and data from the devices needing visibility into the connected SANs, so the FSPF database is negligibly increased.

The iSCSI standard was designed to replace FC with Ethernet as the principal SAN storage interconnect. It was thought an early iSCSI application would be storage-to-storage replication. But market realities killed that. EMC and HDS use mirroring, snapshot and replication protocols that aren't compatible with iSCSI, but are with FCIP, iFCP and TCP/IP in general.

WAN throughput enhancers
The value proposition for storage mirroring and replication over TCP/IP is, of course, the reduction of ongoing operating expenses. However, TCP/IP is notoriously inefficient for storage traffic and throughput. This has led some vendors to implement performance-enhancement proxies to enhance throughput for storage replication over TCP/IP solutions. The following products all have various degrees of value to the users in helping fulfill the promise of storage replication over TCP/IP.

EMC SRDF/A for Symmetrix: EMC has enhanced SRDF with the asynchronous release. This version of SRDF disk replication software reduces bandwidth consumption by 30% because it mirrors delta sets of data every 30 seconds instead of constantly updating ordered writes (as the traditional synchronous version of SRDF does). This should alleviate some of the performance issues of the GigE director, but it doesn't go far enough. SRDF/A throughput still declines to unacceptable levels when the distance is above approximately 500 miles and especially when there are bit errors or line congestion.

Cisco Fast Step for FCIP on the MDS IP services blade: Cisco has implemented a modified version of TCP on its FCIP IP services blade in its MDS SAN switch line. Fast is designed to eliminate much of the saw- tooth effect of TCP when it encounters congestion or bit errors. Standard TCP will drop the packet size immediately back to the minimum of 64 bytes until the congestion goes away or the bit errors decline. The cycling back to maximum packet size can be quite slow.

Cisco Fast solves this problem with two enhancements. First, it drops back more slowly to bigger packet sizes such as 512 bytes, 256 bytes and so on, and returns to the maximum speed faster. This enhancement can increase line utilization as high as 95%. These test results are on clean lines or low bit error rates as seen in the testing at Sprint. If the bit error rates are high, then the results decline significantly to an average of 50% or less. The downside is that it tends to be pricey. It costs approximately $35,000 for the blade, plus the cost of the director at around $2,500 per port.

McData fast writes for the IPS 3300, 4300 and Eclipse 1620: McData implements a fast write enhancement for iFCP. This performance enhancement speeds up mirroring operations. Test results at HDS and EMC have shown twice the IOPS as storage over IP WAN. It has limited impact on snapshots or bulk copies, and no significant line utilization gains. Distance and bit errors negatively impact throughput. For the bulk data traffic, McData implements compression. Testing from EMC and HDS showed that distance and line bit errors reduced throughput.

One interesting advantage of iFCP is its ability to limit disruptive data traffic flow between SANs. It acts like a firewall between SAN islands only allowing the desired data to move back and forth. Although this has little impact on raw data performance, it has a huge impact on the general performance of SAN fabrics themselves.

CNT's IP over UDP: CNT is the granddad of storage over IP. The company has significant experience with ESCON and FC over IP. It doesn't utilize true Transmission Control Protocol (TCP). Instead, CNT leverages its patent pending 32-bit Cyclic Redundancy Checking (CRC) on top of User Datagram Protocol (UDP) and adds compression on top of that. This gives it guaranteed delivery on the faster UDP. It also stacks SCSI commands called Command Tag Queuing. This eliminates multiple trips across the WAN link by bunching sequential SCSI commands together and sending them all at once.

Because CNT has the largest number of installations, results are available and a tad disappointing. User and some vendor testing have shown results ranging from 35% to 80% line utilization. The results tend to be better with lower bandwidth (sub DS3 speeds.) As distance increased, performance declined.

NetEx HyperIP: HyperIP is an intriguing solution because it is purely software. It is an RFC 3135 compliant TCP/IP performance enhancing proxy. Hyper IP runs on Linux OS and Intel hardware. It's a standardized TCP/IP node on the network requiring no modifications to LAN/WAN infrastructures and no proprietary hardware. The results of testing at EMC, Telstra and NTT have been nothing short of astonishing.

EMC saw SRDF achieving 90% and greater bandwidth utilization from distances of hundreds of miles (with high bit error rates on dirty lines) to as far as geosynchronous satellite distances (something SRDF was never designed to do). EMC tested throughput to as high as 450Mb/s. All of these results were without HyperIP compression turned on. Telstra's testing of HyperIP with Veritas Volume Replicator consistently saw results that doubled the throughput of the native product, regardless of distance.

HyperIP eliminates or reduces TCP/IP latency issues by terminating TCP at either end. Interestingly, the utilization test results were statistically the same with and without high bit error rates on the line. With compression turned on, testing showed line utilization rates averaging approximately 200% on DS3 to OC3 lines. The higher the capacity of the line, the less impact the compression had. OC48 lines showed no gain from the compression.

Overall, the results are impressive, but this solution is limited to storage replication solutions that run natively on TCP/IP and Ethernet. HyperIP doesn't support FCIP, iFCP or other FC SAN over WAN solutions.

Storage replication has become a popular implementation for enterprise environments because of its simplicity and speed in disaster recovery. Most of the implementations are being made over TCP/IP networks to save both capital expenditures and operating expenses, but TCP/IP is notoriously inefficient.

But one size doesn't fit all. Each enhancement has its advantages and disadvantages. The TCP/IP enhancement that seems to have the most bang for the buck appears to be NetEx's HyperIP. Of course, it's important to test any of these enhancements in your environment before buying.

This was first published in February 2004



Find more PRO+ content and other member only offers, here.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:



  • Tintri VMstore T5000

    Like all of its VM-aware storage systems, Tintri’s first all-flash array -- the Tintri VMstore T5000 -- allows admins to bypass ...

  • SolidFire SF9605

    The high-capacity SolidFire SF9605 uses SolidFire’s Element OS 8 (Oxygen) to deliver new enterprise features such as synchronous ...

  • HPE 3PAR StoreServ 20850

    HPE 3PAR StoreServ 20850 holds 1,024 solid-state drives (SSDs). Hewlett Packard Enterprise claims it can deliver more than three ...





  • Asigra Cloud Backup Version 13

    Asigra Cloud Backup Version 13 provides an AWS Elastic Block Store Snapshot Manager and the ability to support Docker container ...

  • Veeam Availability Suite v8

    Veeam Availability Suite v8 offers several key backup software components in one package, including Veeam Cloud Connect, Snapshot...

  • Druva inSync 5.5

    Druva inSync 5.5 endpoint backup software stands out with its proactive compliance, cloud app integration, full text search and ...