Problem solve Get help with specific problems with your technologies, process and projects.

DWDM, SONET/SDH or IP based technologies for SAN extension?

Storage expert Greg Schulz offers advice on how to choose the right SAN extension technology for your environment.

"How to compare FC and SCSI" was excellent. I was wondering how to compare SAN extension technologies and in particular when should you use DWDM, SONET/SDH or IP based SAN extensions?"
To pick a data transport to fit your needs, first consider your storage requirements:

  • Distance. How far away do you need to keep a copy of your data?
  • Bandwidth. How much data must be moved and in what timeframe?
  • Recovery point objective. Can you afford data loss? At what point do you need to recover data from?
  • Recovery time objective. How quickly do you need to recover?
  • Latency. What are your applications' response time requirements?
  • Storage-over-distance options:

  • Dedicated and dark fiber optic cabling and wavelength services
  • Wave division multiplexing (WDM, CWDM, DWDM)
  • SONET/SDH optical carrier (OC) based networking and packet over SONET (POS)
  • Metropolitan and wide area Ethernet services
  • TCP/IP based services, networks and protocols including FCIP, iFCP, iSCSI
  • For short distances (35-80 km) using dedicated dark fiber optic cabling, ultra-long range GBICs and SFPs can be used. These include wavelength specific CWDM GBICs and SFPs to support Fibre Channel (FC), FICON and Ethernet networks.

    IP network services are readily available and can be used in conjunction with FCIP (FC over IP), iFCP and iSCSI for distance-based storage applications. You can improve effective TCP bandwidth to support remote mirroring and other data movement functions using bandwidth acceleration and optimization technologies from vendors such as NetEx and Peribit (now Juniper).

    Optical networks including CWDM, DWDM and SONET/SDH are well suited for low latency applications that require congestion free bandwidth. Another benefit of optical based networks is the ability to support different network speeds (1 Gbps to 10 Gbps including sub-bit rate) and interfaces (FC, FICON, ESCON, Ethernet) among others). Sub-bit rate network interfaces include ESCON (17 MBps) and 10/100 Ethernet (10 Mbps and 100 Mbps) that operate below higher speed interfaces such as 1 Gbps.

    SONET/SDH can be used for very short distances in metropolitan areas where dedicated fiber optics and DWDM are not available, as well as for national and international applications. SONET/SDH was designed to be a metropolitan area network (MAN) and wide area network (WAN) backbone supporting interoperability amongst carriers and service providers with very good resiliency and guaranteed bandwidth. You can use as a general guide for SONET/SDH latency delay of about 1 millisecond (ms) per 100 route miles (one-way) regardless of the bandwidth level. A route mile refers to the actual path the optical network takes between two points. This will rarely be a straight line and for MANs and WANs, can involve paths between different city pairs. Check with your bandwidth service providers as to what the route path and network latency will be for your specific application and get a service level agreement for that level of service.

    Using time division multiplexing (TDM) aggregation multiple slower network interfaces can share higher speed network bandwidth. This enables slower networks to share the higher bandwidth without having to allocate network circuits and bandwidth on a one for one basis. Aggregation equipment from vendors including Adva, Ciena, Cisco and Nortel can attached to an OC circuit for aggregation of sub-bit rate networks and bandwidth requirements.

    WDM including CWDM and DWDM, along with SONET/SDH including TDM are complimentary technologies that can be part of a total solution. For example using dedicated dark fiber where available for short haul distances (100-200 km or less) for synchronous applications and SONET/SDH for longer distances to support asynchronous applications. TDM can be used to aggregate multiple slower interfaces onto faster DWDM or wavelength service, as well as onto faster SONET/SDH OC services.

    There are many sources of fiber-based services that vary by geographical location, some being regional and others being international. Check with service providers to verify what specific services they support and in what locations. When talking with prospective fiber optic bandwidth service providers ask about estimated and measured latency, as well as service level agreements between locations. If you are looking at wavelength services, check to see if there are any restrictions of the service provider on using TDM equipment for aggregation of slower and under utilized interfaces.

    Work with your network bandwidth provider's storage partners including EMC, HDS, HP, IBM and Sun/STK among others to verify support and applicable certification for storage over distance applications and solutions. Pay close attention to details including protected or unprotected services, diverse paths, problem resolution and network latency. For storage applications, low latency can be as equally important as the amount of bandwidth, particularly for time sensitive, synchronous applications.

    You can read more about storage over distance topics by Greg Schulz in the Storage magazine article "Bridging the gap", SearchStorage Tip "Bridging the gap" and in Chapter 6 "Metropolitan and Wide Area Storage Networking" in the book "Resilient Storage Networks" (Elsevier).

    Dig Deeper on SAN technology and arrays