This article can also be found in the Premium Editorial Download "Storage magazine: The best high-end storage arrays of 2005."
Download it now to read this article plus other related content.
IP network technology gets a lot of attention, but most storage is sent over optical networks, which are well suited for applications that require low latency and congestion-free bandwidth. (Note that in the networking world, bandwidth is typically referenced in bits per second; for example,
one billion bits per second or 1Gb/sec. Storage is referenced in bytes per second; for example, 100MB/sec.)
Dedicated fiber: Dark fiber refers to a dedicated long-range, single-mode, fiber-optic cable pair (send and receive) that exists between two or more points. A dark fiber cable can be purchased or rented from a carrier or service provider. Dark fiber can create a private network supporting network interfaces, including ESCON, Ethernet, Fibre Channel (FC), FICON and synchronous optical network/synchronous digital hierarchy (SONET/SDH), or can be used with wave division multiplexing (WDM). With dark fiber, it's the responsibility of the user or service provider to light and manage the fiber-optic cable, including the creation of redundant paths and traffic management. For relatively short distances (35km to 80km), ultra-long-range gigabit interface converter (GBIC) and small form-factor pluggable (SFP) optical transceivers coupled with SMF optic cable can be deployed. Dedicated, dark fiber optics can be difficult and expensive to obtain outside of a user's premise in MAN and WAN environments vs. shared-bandwidth-based options.
Wave division multiplexing: Typical fiber-optic deployments involve a fiber-optic cable that's dedicated to a specific network interface. Using WDM, a fiber-optic cable can support multiple network interfaces, each with its own private bandwidth. WDM technology sends light through an optical device using different lambdas (wavelengths) of light. Each network interface that's attached to a WDM-based wavelength functions as though it has its own virtual fiber-optic cable. Consequently, the performance of the different networks varies and each network can be dedicated to support applications with different bandwidth requirements. In addition to supporting simultaneous networks, WDM enables longer distances using long-range optical transceivers.
There are two variations of WDM and they differ in price, capacity, distance and management capabilities. The first is coarse wave division multiplexing (CWDM), which has a lower cost, lower bandwidth and spans shorter distances (80km to 100km). The other variation is dense wavelength division multiplexing (DWDM), which provides more interfaces and greater bandwidth, as well as support for longer distances at a higher price. CWDM is a good entry-level, low-cost solution for applications that have lower bandwidth and distance requirements. DWDM can support storage applications such as synchronous data mirroring and replication over distances of 300km or more; but as distances increase, so does latency.
SONET and SDH optical networks: SONET and SDH are optical-based backbone transport networks deployed by carriers and service providers for metropolitan and wide-area networking on a global basis. SONET and SDH networks support voice, data and video with guaranteed bandwidth, low latency over long distances (thousands of kilometers) and high availability using redundant network paths (rings). SONET (in the United States and some other countries) and SDH (in Europe and other parts of the world) are often referred to as SONET/SDH. SONET/SDH was designed to be a wide-area networking transport providing resilient, low-latency, high-bandwidth connections using fiber-optic networks and traditional electrical (copper) cabling. SONET and SDH networks can use WDM as an underlying infrastructure to maximize fiber-optic bandwidth and support additional network services.
While dedicated, dark fiber-optic service and WDM may not be available in all locations, SONET/SDH services are much more accessible--often from multiple service providers. Thus, SONET/SDH can be used as a local, metropolitan and global fiber-optic-based transport network for voice and data. As a network transport, SONET/SDH supports diverse network traffic, including TCP/IP, FC via generic framing protocol, asynchronous transfer mode (ATM) and packet over SONET (POS) among others.
SONET bandwidth is subdivided using time division multiplexing (TDM). SONET network bandwidth is allocated as optical carrier (OC) levels with SDH network bandwidth being allocated as a synchronous transport module (STM) level. OC levels are based upon multiples of 51.840Mb/sec; for example, OC-3 is 3 x 51.840Mb/sec or 155.5Mb/sec. Other bandwidth increments include OC-12 (12 x 51.840Mb/sec) and OC-48 (48 x 51.840Mb/sec). As a general guide, SONET/SDH latency delay is approximately one millisecond per 100 route miles (one-way) regardless of the bandwidth level. A route mile refers to the actual path the optical network takes between two points. Check with your bandwidth service providers as to what the route path and network latency will be for your specific application and get an SLA for that level of service.
Wavelength services: Carriers and service providers offer fiber-optic wavelength services that are a wavelength provisioned from a WDM device. A wavelength service is similar to a dark fiber, and can be used to span distances for private LAN and SAN network interfaces at various speeds. As opposed to dark fiber, with a wavelength service you have a private wavelength isolated, but share the physical cable with others. Wavelengths offer the fastest connection speeds, lowest latency and most effective bandwidth alternative when dedicated dark fiber isn't available. A wavelength service can be dedicated to a particular network transport (ESCON, Ethernet, FC or FICON) or for shared access using a SONET/SDH gateway. Wavelength services are considered managed solutions, and can be protected (built-in redundancy) or unprotected (no alternate paths). Users can attach their LAN and SAN network devices, including switches and routers, to the wavelength fiber-optic connection.
TCP/IP (IP): TCP/IP (IP) utilizes different transport networks and services, including SONET/SDH, WDM, and Ethernet on copper and fiber-optic cabling. Storage over IP can involve many different protocols and technologies, including NAS for file-based access of data, iSCSI for block data access, as well as FC transported over IP using iFCP or FCIP for long-distance SAN extension and remote mirroring.
IP is a good, general-purpose network protocol for storage apps that aren't sensitive to latency or that have lower bandwidth requirements, and for cost-sensitive storage environments. This isn't to say IP doesn't support high bandwidth--quite the contrary. IP supports high bandwidth when using fast network transports, including SONET/SDH, DWDM and carrier Ethernet. A common mistake is to look at dollar per Gb/sec and assume full bandwidth will be achievable. This is similar to evaluating disk storage simply on a dollar-per-gigabyte basis instead of a more realistic dollar per usable, effective gigabyte. When making comparisons among bandwidth speeds, an important number to compare is the packet delivery ratio (PDR). As more layers are involved in the movement of data, more latency will be introduced. The PDR number describes how high or low the packet retransmission and packet loss is, an indicator of effectiveness and latency. For storage apps, low latency can be as important as the amount of effective available bandwidth, particularly for time-sensitive, synchronous applications (see "Transport options: Pros and cons," this page).
|Transport options: Pros and cons|
Layering and latency
Remember that the more layers and protocols used in the network, the longer the latency and potential disruption to the storage applications. For example, on the left-hand side of the "Bandwidth layers" diagram, an IP-based network may have a lower cost for a specified amount of bandwidth. However, when looking at the multiple layers involved (FC mapped onto IP using FCIP, while the TCP and IP layers are mapped onto SONET), the latency adds up; depending upon the design of the network, congestion could reduce the effective bandwidth. Moving to the right on the diagram, an example of FC extension using ATM/POS mapped to SONET is shown, followed by FC mapped to SONET with less latency and layers, followed by FC on dedicated fiber optic (WDM). Understanding what layers are involved in the network is important because each layer adds complexity, cost and latency.
Reliability, availability and serviceability
Diverse network paths are critical for uninterrupted network service. Make sure your network provider can guarantee diverse network paths not only through its core networks, but its partners' core networks. Secondly, determine how the service provider will manage and guarantee network performance (low latency and effective bandwidth). It's possible that different network bandwidth providers could use the same network route paths. Fiber-optic bandwidth services, including wavelength services, can be protected or unprotected. With a protected optical or wavelength service, the service provider uses dual paths through its network. A protected bandwidth service may have a handoff (connection) to the user of a single or dual pair of fiber-optic cables depending on service type. Another alternative is to use self protection, where two diverse wavelengths of fiber-optic services are used from the same or different providers, each with a diverse path. In this scenario, the customer takes responsibility for integrating the fiber-optic network into their environment.
This was first published in August 2005