This article can also be found in the Premium Editorial Download "Storage magazine: Tips for unifying storage management."
Download it now to read this article plus other related content.
Declining bandwidth costs
The cost of WAN connections has declined over the past few years. In some cases, the declines have been as much as 75% over the last two years. Expectations are that bandwidth costs will continue to decline (albeit at a slower rate) because the supply continues to exceed demand.
In spite of this ongoing trend, large bandwidth, DS3, OC3, OC12, OC48 and OC192 are still expensive. Storage replication requires large amounts of bandwidth. Shared or dedicated bandwidth makes no difference. Just calculating the bandwidth makes this clear. For example, replicating 1TB of storage would take nearly 50 hours over a DS3, assuming 100% bandwidth utilization. Using the same assumption, it would take approximately 14.4 hours for an OC3 and 3.6 hours for an OC12. In the real world, 100% bandwidth utilization isn't likely. The reason is TCP/IP, which is the preferred storage-to-storage WAN protocol of choice. There are three reasons for this:
- There's a perception that the bandwidth is free because the TCP/IP WAN links already exist for interactive traffic. The conventional rationale is that storage-to-storage replication, or snapshot, typically occurs at night or on weekends when the majority of users are not utilizing the network. This allows the already existing TCP/IP WAN links to be leveraged by the storage applications without negatively affecting other applications.
Storage over IP
One of the early applications for storage over IP is mirroring. The three IP standards that have emerged are Fibre Channel over IP (FCIP) Internet Fibre Channel Protocol (iFCP) and iSCSI.
FCIP is an Internet Engineering Task Force (IETF) standard that encapsulates FC frames in TCP/IP packets and places them in GigE frames. It's used to connect FC SANs over WANs for data replication.
The iFCP standard assigns a TCP/IP address for each FC frame and also packages them typically in GigE frames; currently only McData offers this. The key advantage for iFCP over FCIP is the ability to selectively move data from specific devices between the SANs.
In other words, a Register State Change Notification (RSCN) fabric disruption on one SAN disrupts the fabric on the connected SAN. The RSCN broadcast is the FC switch methodology for informing all devices on the fabric that a change has occurred. Larger fabrics contain more RSCNs, which disrupt data flows. Isolating interconnect SAN RSCNs from one another reduces the impact on the fabric and improves performance.
Fabric Shortest Path First (FSPF) is the FC fabric standard routing algorithm. The FSPF database size is correlated to the number of devices, paths, switches and ISLs in a SAN fabric. When FCIP interconnects two or more SANs into one, this increases the FSPF database for all switches, slowing performance. When iFCP interconnects two or more SANs, it only passes the address and data from the devices needing visibility into the connected SANs, so the FSPF database is negligibly increased.
The iSCSI standard was designed to replace FC with Ethernet as the principal SAN storage interconnect. It was thought an early iSCSI application would be storage-to-storage replication. But market realities killed that. EMC and HDS use mirroring, snapshot and replication protocols that aren't compatible with iSCSI, but are with FCIP, iFCP and TCP/IP in general.
This was first published in February 2004