Published: 02 Jul 2012
10 Gbps Ethernet is here, with plenty of products available, so it’s only a matter of time before most data storage shops decide to step up from 1 GigE to 10 GigE.
Many data centers still use Gigabit Ethernet (1 Gbps) -- or 1 GigE -- to handle normal local-area network (LAN) traffic between laptop computers, desktop computers, file servers and application servers. But many application and file servers currently use multiple 1 GigE network interface cards (NICs) because they need more total bandwidth. It’s not uncommon to see four, six or eight 1 GigE ports in these servers. Some of the ports are used for management functions, while others are dedicated to network traffic flow, including some file or block storage traffic. Certainly, with the increasing adoption of virtualization, we find that more network bandwidth is needed.
The solution has actually been lurking deep inside the data center for a few years: 10 Gigabit Ethernet (10GigE). This technology, as its name implies, is 10 times faster than Gigabit Ethernet, and was actually ratified as a standard back in 2002. However, its price point has been too high for a very long time, so adoption during its first few years was limited to data center core networking, where the higher bandwidth and price could be justified. Initially, it was used primarily for switch-to-switch connections for trunking of network connections. Earlier implementations of 10 GigE also used different cabling and connector technologies requiring fiber-optic or specialized copper cabling. For several years, 10 GigE couldn’t use the same type of familiar cabling we use for Gigabit Ethernet and the slower speeds of Ethernet.
Growing use of 10 GigE
In the last few years, however, 10 GigE has been increasingly deployed not only at the network switch but within application servers. Although it’s still somewhat expensive, the price per port for 10 GigE technology has dropped enough to make it viable for more organizations. Many of the blade servers introduced in the last two or three years have two or more 10 GigE ports on the blade chassis that allow for increased total bandwidth into and out of the blade chassis. More recently, 10GBASE-T was introduced, which makes 10 GigE technology available with cabling and connectors that are very similar to the familiar Cat 5e and Cat 6 cables we use today for Gigabit Ethernet.
Servers, solid-state spur adoption
In early 2012, another event occurred that’s expected to stimulate the adoption of 10 GigE. The major server vendors introduced new servers built around Intel’s newest server processors (Intel Xeon E5 series, formerly known as “Romley”) that have options for 10 GigE connections directly on the motherboard or a nearby daughterboard. By having 10 GigE as a factory option on individual servers, the transition to 10 GigE is expected to be easier and more appealing to a larger number of organizations.
In addition to server virtualization driving the need for more network bandwidth, solid-state storage or solid-state drive (SSD) technology is also driving up CPU and network utilization. As SSDs in various form factors are deployed more widely, storage performance (typically among the slowest performing parts of an IT infrastructure) is increasing, which has a ripple effect on other technologies. We’ve observed in our Demartek Labs testing that SSDs and 10 GigE go well together.
Due to all these factors, some industry analysts who observe the networking market are making some pretty strong predictions about the growth of the 10 GigE market over the next three or four years. At Demartek Labs, we’re conducting an increasing number of server and storage hardware and software tests that require or prefer 10 GigE.
10 GigE ports
One of the advantages 10 GigE has over 1 GigE is that 10x more bandwidth can be achieved in the same form factor. This applies to switch ports and host adapter ports. In a host server, for example, if eight 1 GigE NIC ports are required, this can be achieved by using two quad-port, four dual-port or eight single-port NICs. This consumes either two, four or eight I/O slots in the server and results in eight cables running from the server. This may work for some environments, but not all servers have that many slots or you may want to use some of the server I/O slots for other purposes, such as RAID controllers, Fibre Channel host bus adapters (HBAs), PCI Express (PCIe) SSDs and so on. Using 10 GigE, one dual-port 10 GigE NIC could be used and it would provide more than the bandwidth of eight 1 GigE NIC ports on a single port with still another port for failover or other management purposes. This would consume only one I/O slot, freeing up I/O slots and potentially consuming less electrical power. It would also require no more than two cables. Alternatively, the same dual-port 10 GigE NIC could be placed into a smaller form factor server, such as a 1U server with only a couple of slots, while still providing plenty of network bandwidth.
A dual-port 10 GigE NIC requires a PCIe 2.0 x8 slot to ensure maximum bandwidth at full duplex operation. Some dual-port 10 GigE NICs support PCIe 1.0 x16 slots. A single-port 10 GigE NIC can work in a PCIe 1.0 x8 slot or a PCIe 2.0 x4 slot. PCIe 3.0 slots aren’t required for a dual-port 10 GigE NIC, and true PCIe 3.0 Ethernet adapters won’t be available until probably late 2012 or 2013.
10 GigE adapters
Additional features are available with 10 GigE adapters. There are versions of these adapters from several adapter vendors that have TCP/IP offload engines (TOE), iSCSI offload, Fibre Channel over Ethernet (FCoE) or all three. These can improve performance in some instances, and in most cases reduce the host CPU utilization by offloading these functions onto the adapter. The adapters that support TCP/IP, iSCSI offload and FCoE together are known as converged network adapters (CNAs) and provide an interesting blend of traffic on the same wire at the same time. This function requires Data Center Bridging (DCB) in the switch to operate. Demartek Labs has published plenty about how this works, as well as the results of some of our lab results. Some performance reports are available on Demartek’s FCoE Zone website.
Most of the Ethernet switch vendors provide 10 GigE switches with a variety of port counts and some with support for DCB. Some of these switches have ports than can support both 1 GigE and 10 GigE, provided the same form factor of connector is used for both. If your switches have ports that support both speeds with the appropriate transceivers or connectors, this can help smooth the transition to 10 GigE.
Connectors and cables
There are basically two common styles of connectors and cables used for 10 GigE. The most common today is the SFP+ connector, which can be used with fiber-optic cables or copper cables. For fiber-optic cables, these are known as OM3, OM4 or OS1. These days, OM4 is the recommended multi-mode fiber-optic cable for 10 GigE, but OM3 is generally considered to be a minimum requirement. OS1 is used for very long distances, up to 10 km or 6.2 miles. SFP+ connectors are also available on copper cables with the distance limited to within a rack or a nearby rack.
The other 10 GigE form factor that’s beginning to gain a great deal of attention is 10GBASE-T. This style of cable isn’t directly compatible with SFP+ but uses the familiar RJ45 plug on the cable. The recommendation for 10GBASE-T is to use Cat 6a or Cat 7 cable, but Cat 6 can work for distances up to 55 meters. Cat 5e cable isn’t recommended for 10 GigE. Adapter vendors are making models of their 10 GigE adapters with either SFP+ or 10GBASE-T ports.
The majority of 10 GigE switches support SFP+ connectors and transceivers, but we’re seeing a number of 10 GigE switches that support 10GBASE-T. The 10GBASE-T products have been a few years behind the SFP+ products, but the 10GBASE-T form factor is expected to gain acceptance more quickly. Some switches are becoming available that have SFP+ ports and 10GBASE-T ports.
A wider wire
10 GigE promises to deliver a lot more networking “punch” while consuming fewer server I/O slots and less cabling. 10 GigE switches provide the opportunity to consolidate networks into simpler infrastructures and network topologies because a large number of 1 GigE ports can be combined into 10 GigE ports while providing more total bandwidth.
BIO: Dennis Martin has been working in the IT industry since 1980, and is the founder and president of Demartek, a computer industry analyst organiz