This article can also be found in the Premium Editorial Download "Storage magazine: Big 3 backup apps adapt to disk."
Download it now to read this article plus other related content.
To appreciate what InfiniBand brings to enterprise storage, you should know a little bit about the technology. IB was originally developed to replace the aging shared-bus paradigm in servers (for example, PCI slots) with a high-speed, low-latency switched serial I/O interconnection. A fabric is created by installing an IB host channel adapter or host target adapter card into each device, and then interconnecting the devices through an IB switch. Advances in silicon now allow native IB ports to be implemented directly in devices and motherboards, eliminating the need for standalone expansion cards.
|Basic InfiniBand channels and data rates|
IB bandwidth can potentially dwarf other network technologies like 4Gb/sec FC or even 10Gb Ethernet (10GbE). IB is based on 2.5Gb/sec bi-directional serial channels that can support point-to-point connectivity between devices on the fabric. Channels can also be aggregated for vastly improved throughput. For example, a typical four-channel IB interconnect can offer throughput up to 10Gb/sec. Channels can also be operated at single, double and quadruple data rates for even more throughput (see "Basic InfiniBand channels and data rates," this page). While 10Gb/sec is common today, 20Gb/sec to 30Gb/sec deployments are available, and the IB roadmap suggests speeds to 120Gb/sec over copper in 2007.
But the appeal of IB is more than just speed. These speeds are achieved with very low latencies of just a few microseconds--a fraction of the latencies found in Ethernet networks. IB also supports a "switched" fabric, so many IB devices can share the same I/O, allowing huge numbers of devices to share the fabric. An example of this scalability is the 4,500-node InfiniBand cluster recently deployed by Sandia National Laboratories.
Another key advantage with IB is its ability to handle memory-to-memory data transfers between servers using Remote Direct Memory Access (RDMA). With RDMA, one server can "look into" another server and read just a few bytes (or the entire memory contents) without the direct intervention of either server's CPU. This frees processing power to continue with more compute-intensive tasks. Although RDMA technology will soon be integrated into 10GbE as the Internet Wide Area RDMA Protocol (iWARP), RDMA is native to the current InfiniBand specification. IB storage systems can easily leverage RDMA by treating the storage device as a server--passing data between server memory and a storage system's internal cache.
This was first published in April 2006