This article can also be found in the Premium Editorial Download "Storage magazine: Storage magazine's data storage products of the year."
Download it now to read this article plus other related content.
By Christine Cignoli
It's barely even here yet, but 10Gb Ethernet (10GigE) is going to have a hard time living up to its hype. Hailed as a "game changing" technology by some, it carries the burden of being a cure-all for storage (and network) managers' problems. But when you look beyond the hyperbole surrounding 10GigE, you'll see the technology is, in many ways, still just emerging.
That's not to suggest that 10GigE won't deliver on its promise -- perhaps not a cure-all, but certainly destined to give iSCSI storage and Fibre Channel over Ethernet (FCoE) topologies a big boost. Still, real products are few and far between at this time, and per-port prices are still at a very un-Ethernet premium. While there have been some early adopters, its first few inroads into the market have been in higher-end implementations such as super-high-performance computing.
Switch vendors are starting down the 10GbE road with recent product releases and certifications. But it may still take some time for 10GbE to hit the mainstream. Brad Booth, chairman of the board of the Ethernet Alliance, thinks 2012 is when costs will be low enough for the market to see widespread adoption. "Generally when a standard is written, it takes about 10 years before it really hits what we consider the big volume adoption," he says. Booth calls those 2012 adopters "tier 3," and says that quicker-adopting tier 2 data centers are probably looking at 10Gb now and considering which cabling they'll choose. He cites Google as a tier 1 data center; they're now running a brand-new data center with 10GbE throughout, according to Booth.
|10Gig rides on copper or fiber|
Optical fiber: The earliest 10Gb Ethernet components were built around optical fiber, but copper cable-based products soon followed. In most data centers, copper is the standard transport for data and storage networks because it's relatively cheap and easy to install. But copper cabling typically can't carry a signal over long distances, so it's used primarily within buildings, while optical cabling is usually used to link facilities over longer distances.
There are a number of IEEE standards for optical cabling with 10Gb Ethernet, each designated by a unique identifier. For example, 10GBASE-SR (short distance) is the standard for the cheapest optical implementation that uses standard optical cabling; it can cover distances of approximately 30 meters to 90 meters. The next step up in price and distance is 10GBASE-LR (long range), which can carry signals up to 10km. There are several additional standards for 10Gig over optics that offer greater ranges.
Copper connections: For copper cabling, 10GBASE-CX4 is an IEEE standard approved approximately five years ago that uses familiar twin-axial cable. Also referred to as 802.3ak, this standard is based on cabling and connectors used for InfiniBand, although some reengineering was done so they're not identical. 10GBASE-CX4 is effective at distances up to 15 meters.
10GBASE-T, approved in 2006, is likely to gain widespread popularity. It uses unshielded (or shielded) twisted-pair cables and will work at up to 100 meters; companies may opt to use already installed Cat 6 cabling, but the distance with be effectively halved. Connectors for 10GBASE-T are familiar, too, RJ-45-style connectors rated at 650 MHz.
One research institute has already moved to 10GbE. Bruce Allen, director of the Hannover, Germany-based Max Planck Institute for Gravitational Physics, chose a Woven Systems Inc. switch as part of their early 2008 systems upgrade. "I think 10Gb is now a proven technology," he says. And being an early adopter is standard for Max Planck. "We're a research institute," he says. "The one guaranteed way to fail is to wait. The wrong decision is waiting." The institute stores and analyzes data, with approximately 1.5 petabyte (PB) currently managed. Allen says storage is secondary for them, with compute cycles the primary goal. "I wanted wirespeed nonblocking to all nodes, and that meant Ethernet," he says.
Woven is one of the few vendors offering 10Gb switches, along with others including, Cisco Systems Inc., Force10 Networks Inc., Foundry Networks Inc. (now part of Brocade) and Mellanox Technologies Inc. Joseph Ammirato, vice president of marketing at Woven, thinks 10GbE will be more affordable by 2010. And, for more widespread adoption, "we need 10GbE as a default feature on server motherboards," he says. "Users still have to buy it as a separate add-on." He targets mid-2010 for 10Gb to be built in.
Ammirato names several customer requirements he thinks will drive 10Gb adoption: consolidation, server virtualization and what he calls desktop virtualization -- a need for organizations to better control mission-critical applications, running them from the data center and not user desktops.
Mellanox is trying to move into the 10GbE market from its base of InfiniBand products. T.A. Ramanujam, Mellanox's senior product marketing manager, says that in the past few years, Ethernet has come into the spotlight as a consolidation or unifying data center fabric. "To accomplish that," he says, "they've taken features from technologies like Fibre Channel and InfiniBand and added that to [a] new Ethernet specification, what's going to be called data center Ethernet or converged enhanced Ethernet."
|A sampler of 10GBASE-T NICs|
Early on, networking vendors were challenged by issues such as power consumption and overheating with their designs for copper-based interface cards for 10Gig Ethernet. Many of these issues have been resolved. Today, there are a number of vendors offering 10Gig Ethernet network interface cards (NICs) that use the 10GBASE-T that supports twisted-pair cables and RJ-45 jacks, including the following:
Ramanujam says per-port prices are approximately $300 to $400 for a 10GbE switch, and predicts that the market will start adopting 10GbE when its price per-port cost drops closer to Gigabit Ethernet's cost. "It's not very far-fetched," he says, estimating that the second half of 2010 will bring 10GbE on the motherboard along with more affordable switches. "By then, the price of 10gig switches will be in the region of $100, $150 per port."
It also remains to be seen how FCoE, a potential 10Gb competitor, will affect 10GbE's adoption and pricing. Allen at the Max Planck Institute thinks 10Gb will triumph. "I tend to think that Fibre Channel is one of those technologies that doesn't really have a good place in the future," he says. "You're better off sticking to commodity stuff." But storage users managing the deeply ingrained FC infrastructures of many businesses might disagree.
In the meantime, the Ethernet Alliance's Booth says the group is busy putting on interoperability demonstrations and educating possible 10GbE users on what they'll need to do to be ready for the transition -- namely, consider their current server utilization and decide on cabling. "Looking at a server running 1gig, do they want to upgrade that server by putting in a new NIC and upgrade it to run 10gig?" says Booth. "Or do they wish to put a better server in there and put this 10gig card in it?"
Users must also think about whether they can reuse their current optical data center cable, or whether they'll need or want to replace copper cabling. Allen says the one mistake the Max Planck Institute made in setting up 10GbE was its cabling choice. They're using copper 10Gb cables, which Allen calls "bulky and inflexible." For $50 more apiece, he says, they could have had optical cables.
10Gb also signals a shift in thinking toward consolidation and centralization. "Unlike previous Ethernet technologies, where primarily volume went to desktop," says Booth, "now most of the volume of 10gig is in the data center, into the core of network."
BIO: Christine Cignoli is the Associate Site Editor for SearchStorage.com and SearchStorage.co.UK.
This was first published in February 2009