Which transports and protocols should you use in your SAN? How do you "size" your SAN in terms of its speed and latency requirements? Below, I've offered a few suggestions on when to use iSCSI, when to use Fibre Channel and when to use something else.
There are some basic factors to consider in determining which protocols to use in your SAN.
- Multipathing. The main determination to make is if you want simple failover for business continuity purposes or if you would like load-balancing as well. You can find ways to get the first for free by using MPIO for iSCSI or failover drivers from your Fibre Channel host bus adapter (HBA) vendor. The latter is a per host option you can purchase from your storage vendor.
- Latency, throughput and distance are going to be the key factors in determining what kind of storage to adopt. If you go through the trouble of building a storage solution that offers enormous IOPS and throughput, you aren't likely to want to bottleneck all of this over a 1 Gigabit Ethernet high-latency iSCSI link. If you are simply provisioning files, there's no need to incur the cost of Fibre Channel. Ideally, leveraging both iSCSI and Fibre Channel is the best option. In some cases you can do iSER (iSCSI over InfiniBand) or iSCSI over 10 Gigabit Ethernet.
ISCSI protocol
ISCSI is a protocol that allows clients, called initiators, to send SCSI commands to SCSI storage devices (targets) on remote servers. It also supports multipathing.
The pros of iSCSI: Unlike Fibre Channel, which requires special-purpose cabling, iSCSI can be run over long distances using existing network infrastructure. You can implement iSCSI over 1 Gigabit Ethernet, 10 Gigabit Ethernet or a specialized version called iSER over InfiniBand that can be as fast as 10 Gbit or 20 Gbit. You can also do trunking or IP bonding of iSCSI to gain additional performance capabilities.
The cons of iSCSI: iSCSI initiators allow your network ports on your servers to see the SCSI devices from your iSCSI SAN. CPU overhead is associated with this, unless you are using iSCSI HBAs, which can provide additional benefits, such as booting from your iSCSI SAN.
When to use iSCSI: Depending upon your latency, throughput and budget requirements, iSCSI can be used in any number of applications. It is particularly popular with VMware and other virtual server requirements. A 10 Gigabit Ethernet iSCSI SAN can be created inexpensively, though I suggest Fibre Channel or InfiniBand for database and email applications.
Fibre Channel connection
Fibre Channel is a gigabit-speed technology used for storage networking. It has become the standard connection type for enterprise storage SANs. Despite common connotations of its name, Fibre Channel signaling can run on both twisted pair copper wire and fiber optic cables. In other words, fiber (ending in "er") always denotes an optical connection, whereas Fibre (ending in "re") is always the spelling used in "Fibre Channel" and denotes a physical connection that may or may not be optical. Fibre Channel Protocol (FCP) is a transport protocol (similar to TCP used in IP networks) that transports SCSI commands over Fibre Channel networks.
Pros of Fibre Channel: The main advantage of Fibre Channel, which comes in 1 Gbit, 2 Gbit, 4 Gbit and 8 Gbit HBAs, switches and directors (big switches), is that it offers much lower latency and much faster throughput than iSCSI, though iSER over InfiniBand rocks. Fibre Channel HBAs also have less CPU overhead than iSCSI, unless you are using an iSCSI HBA or a TOE network adapter.
Cons of Fibre Channel: This can be the most expensive type of SAN, and HBAs have become a lot lower in price. Fibre Channel has shorter distance limits than iSCSI.
When to use Fibre Channel: Use Fibre Channel for IOPS-demanding applications, or if you have a requirement for really high throughput (MBps).
InfiniBand specification
InfiniBand is a switched fabric communications link used in high-performance computing. Its features include Quality of Service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high-performance I/O nodes, such as storage devices. You can also trunk IB connections.
Pros of InfiniBand: InfiniBand offers low CPU overhead and extremely low latency. It can deliver up to 20 Gbit throughput. It can be lower in cost than Fibre Channel.
Cons of InfiniBand: InfiniBand only covers short distances (15 meters) unless you invest in IB long-range extenders and have fiber or high-end WAN connectivity in place. You also might find limited support for certain applications or hardware.
When to use InfiniBand: When the data has to have the best level of performance possible. Of course, if you are doing HPC (compute clusters), you'll likely be investigating IB. Another consideration is for I/O virtualization, which holds the promise of reducing your network and Fibre Channel SAN infrastructure by using virtual Fibre Channel HBAs and virtual NICs over an IB connection to the server.
If you want a primer on SAN, I'd recommend the book Storage Area Networks for Dummies by Chris Poelker and Alex Nikitin, which provides a practical overview without assuming you know about buffer credits or G_ports or U_ports or oversubscription.
About the author: Joel Lovell is senior storage consultant for Storage Engine Inc. His specialty is high-performance storage and storage consolidation. He is EMC-trained in business continuity solutions, enterprise storage infrastructure and enterprise storage management. He previously was a strategic storage specialist for the Americas for Silicon Graphics and a senior systems engineer for EMC.
