This article can also be found in the Premium Editorial Download "Storage magazine: Tame the e-mail beast: Valuable e-mail management strategies."
Download it now to read this article plus other related content.
InfiniBand is one of the most misunderstood and under-hyped technologies. InfiniBand was developed to eliminate the server I/O bottleneck. I/O has increasingly become the gating factor to increasing server performance. With the recent decisions by Intel and Microsoft to eliminate or reduce their development of InfiniBand products, there's been much confusion as to its market viability. Many in the financial community and industry believe InfiniBand will die because Intel and Microsoft have abandoned InfiniBand, but they're incorrect.
|How to calculate an ROI|
The premises are wrong because neither Intel or Microsoft has abandoned InfiniBand. Intel is still a major player in the InfiniBand Trade Association (IBTA), and is supporting developers' conferences for InfiniBand.
Intel decided not to develop its own InfiniBand silicon because of volume considerations. InfiniBand isn't a desktop or laptop technology, meaning Intel wouldn't have the volumes to justify the development. Microsoft didn't kill InfiniBand - they delayed it until the market emerges.
InfiniBand is a detailed, tight specification for serial I/O. It defines the technology for interconnecting CPUs and I/O nodes to form a fabric that's independent of the host operating system and processor platform. InfiniBand is designed to be a robust low latency architecture (hundreds of nanoseconds) replacement for the aging PCI bus.
InfiniBand utilizes shared memory vs. shared bus that leverages a concept called virtual pipes or lanes. Virtual pipes allow multiple fabrics to coexist on a single topology (see "InfiniBand Server I/O.") Performance definitions include 1X (2.5Gb/s) links, 4X (10Gb/s) links and 12X (30Gb/s) links. The native protocol is VIA (virtual interface architecture) which is an OS bypass utilizing nominal CPU cycles. The network layer is TCP/IP v.6.
InfiniBand is the first simplified I/O fabric designed from the ground up for all aspects of server I/O. Now the servers can be managed, scaled and maintained, without disruptions to either the storage or IP fabrics.
InfiniBand value propositions
1. Reduced capital and operating costs because there are fewer systems and parts purchased, implemented and managed to accomplish all the I/O.
2. Reduced complexity because of the nondisruptive plug-and-play nature of InfiniBand.
3. Reduced management costs because the I/O is managed as a central single entity.
4. Reduced infrastructure costs because there will be fewer Ethernet and FC switch ports.
5. Lower cost server scaling through clustering and lower cost DBMS clustering vs. higher cost SMP(symmetric multiprocessing) or NUMA (non-uniform memory access) machines because of lowest latency network for IPC (interprocess communication).
6. Performance future proofing because it is 10Gb now and is already specified for 30Gb later.
7. Reduced power and real estate costs because the server I/O disaggregation reduces the footprints and power requirements of every server, in addition to the reduction of actual powered equipment.
Reduced capital and operating costs. Disaggregated shared I/O means fewer adapters, NICs, switch ports and fabric switches. This should equate into reduced capital expenditures, but InfiniBand is a new technology that doesn't have the volumes of FC or Ethernet. Even though there's less equipment to buy, the capital savings works out to be negligible. These savings increase dramatically when clustering and/or operating costs are added back in. The value proposition accuracy: medium.
Reduced complexity. Disaggregated shared I/O equals less equipment. Less equipment equals simpler fabrics. The value proposition accuracy: high
Reduced management costs. Disaggregated shared I/O equals less equipment, that means less IT staff time is required to manage the fabrics saving operating costs. The value proposition accuracy: high
Reduced infrastructure costs. Disaggregated shared I/O means fewer cables, cable runs, transceivers and connectors. The value proposition accuracy: high
Lower cost server scaling through clustering and lower cost DBMS clustering. The concept is called scaling out instead of scaling up. SMP and NUMA servers are significantly more complex and expensive than clustered rack or blade servers. For clustered servers and DBMS clusters to be as effective as SMP and NUMA servers, the clustering interconnect must be very low latency and very high bandwidth. InfiniBand meets those requirements. The value proposition accuracy: high
Clear upgrade Path InfiniBand is the only released 10Gb interconnect today (4x), and the only one specified for 30Gb (12x) tomorrow. Value proposition accuracy: high
Multiprotocol, intelligent switches
This latest storage networking technology will allegedly solve all IT data center infrastructure problems. A good way to think of this technology would be director-class SAN switches, carrier class Ethernet switches, storage subsystem controllers and protocol gateways all rolled into one system.
Multiprotocol intelligent switches are similar to directors because they are high port count switches with 32 to 256 ports. Some have specifications of up to 2,400 ports in a single image. They are also similar because they have no single points of failure, have hot swappable parts and provide essentially nondisruptive operations.
One way multiprotocol intelligent switches are different from directors is that they are protocol and topology agnostic. They don't care if the SAN is FC, iSCSI, TCP/IP on GigE, FCIP, iFCP or InfiniBand. They tend to support all or most of them converting one to other and back again at wire speeds. They are also different in that they tend to have storage applications integrated in the switch. These applications typically include some mix of storage virtualization, volume replication, remote mirroring, NAS and application-aware storage. Many of these architectures have processors on every port. This additional functionality comes with a very steep penalty. Higher costs in equipment, maintenance and fabric latency. The proprietary nature of the equipment also typically means a vendor lock out environment.
This was first published in October 2002