What you need to deploy FCoE: A storage checklist

This storage checklist looks at what you need for deploying Fibre Channel over Ethernet (FCoE), including cabling, converged network adapters and FCoE switches.

IT organizations that want to dip their toes into the Fibre Channel over Ethernet (FCoE) waters can start by purchasing converged network adapters (CNAs) and top-of-rack or end-of-row switches that support FCoE. But, they'll have to wait for high-port-count core switches and storage support.

Here's a rundown of the new technology needed for FCoE:

Cabling for FCoE

The RJ45 connectors and CAT-5 or CAT-6 cabling that users now have for traditional 1 Gigabit Ethernet are currently not supported in any Fibre Channel over Ethernet (FCOE) products, according to Stuart Miniman, a technologist in the Office of the CTO at EMC Corp. Users have two options today for FCoE: the optical cabling generally found in Fibre Channel (FC) storage area networks (SANs) or a new type of twinaxial (twinax for short) copper cabling.

More on FCoE

How to get from Fibre Channel to Fibre Channel over Ethernet

What you need to deploy FCoE: A storage Checklist

Enterprise data storage requirements for FCoE

On the plus side, the black twinax cable requires less power and is lower cost, but because the distance is limited to fewer than 10 meters, a user will likely need optical cabling to go from the top-of-rack switches to the LAN.

"Everybody should re-examine their cabling configuration," said Miniman.

He also recommended that users move from the orange OM2 optical cabling to the newer aqua OM3 cabling, which will permit connections over greater distances.

Converged network adapters

Converged network adapters combine the functionality of Ethernet NICs and Fibre Channel host bus adapters (HBAs) in an Fibre Channel over Ethernet environment, enabling users to reduce the number of adapters they need to buy, cut their port count and eliminate a healthy number of cables. Emulex Corp. and QLogic Corp. are the top two CNA makers, and Brocade Communications Systems Inc. is the new kid on the block.

Users can acquire CNAs from resellers or through their enterprise data storage vendors. EMC, for instance, rebrands CNAs from Brocade, Emulex and QLogic, and through its Connectrix product line. NetApp Inc. offers CNAs from Brocade and QLogic.

One key consideration for the earliest adopters has been getting second-generation CNAs. The first-generation CNAs were too large for a number of servers, whereas the single-chip, second-generation CNAs are the size of HBAs and slide easily into server adapter slots.

The Mississippi Department of Information Technology Services found the second-generation cards solid after encountering problems with first-generation CNAs in its FCoE test environment. Kemper Porter, systems manager of the data services division, said the first-generation cards physically didn't fit into the state's IBM Corp. servers.

"I could not close the lid," he said. "We tried all kinds of goofy stuff to try and deal with that -- floppy cables and laying the card on its side. It was not a production environment so you could fool around with it. I found these little ribbon cables you can make that will extend the PCI interface an extra inch and a half, but it did not work well to say the least."

Fibre Channel over Ethernet switches

Users can't go FCoE directly from their servers to their storage arrays, so they must buy switches that support the new technology. For the earliest adopters, that usually means top-of-rack switches from one of the two major vendors, Brocade or Cisco Systems Inc.

Cisco in May 2008 began shipping its top-of-rack Nexus 5000 Series switches. The 56-port Nexus 5020 includes 40 ports that support 10 GigE, Cisco Data Center Ethernet (DCE) and FCoE Small Form Factor Pluggable Plus (SFP+). Two expansion slots can be configured to support up to 12 more of the same type of ports, 16 Fibre Channel switch ports or a combination of both. The Nexus 5010 has half the port count.

The Brocade 8000 provides 24 10 GigE ports for LAN connections and eight FC ports for FC SAN connections, and supports Converged Enhanced Ethernet (CEE), the incarnation of lossless Ethernet that its products support, in contrast to Cisco's DCE. The IEEE standard that both vendors now pledge to support is known as Data Center Bridging (DCB) which remains a work in progress.

The Cisco Nexus 5000 Series and the Brocade 8000 top-of-rack switches separate the Ethernet LAN traffic from the Fibre Channel SAN traffic. The existing FC infrastructure stays intact, running from the core switches to the disk arrays.

"This is the safest way of deploying FCoE and still taking advantage of the cost savings," since most of the benefits emanate from the access layer between the servers and the first-hop switch, said Kash Shaikh, a senior manager of market management for Cisco data center switching. "You don't have to wait for end-to-end FCoE to take advantage of the savings."

Shaikh said more than 900 customers have purchased the Nexus 5000 since it started shipping and 30% of them have bought Fibre Channel over Ethernet licenses for the product.

Another Cisco FCoE option is its Unified Computing System (UCS) which combines computer, network, storage access and virtualization technologies. The networking portion of UCS, which started shipping last summer, is based on the same unified fabric technology as the Nexus switch, according to Shaikh.

Cisco also has announced plans to support FCoE in its Nexus 7000, which Shaikh said was designed to support converged LAN and SAN traffic, and its MDS core switches. Brocade recently made available an FCoE 10-24 blade switch that fits into its DCX backbone switches. The 10 stands for 10 GigE, and the 24 represents the number of enhanced 10 GigE ports in the blade.

"People connect their servers to the blade, and they are in business, because the DCX is already connected to the SAN," said Ahmad Zamer, a senior product manager for new technology protocols at Brocade. "It already has one or two blades that have all the Fibre Channel ports that people need. Nothing else needs to be done."

Users often buy switches through their storage vendors. NetApp, for instance, resells Cisco Nexus switches and, in August 2009, added Brocade switches to its product list. Hewlett-Packard Co. partners with Brocade, Cisco and QLogic for multiprotocol SAN switches. EMC rebrands switches from Brocade and Cisco and sells them through its Connectrix product line.


Dig Deeper on Ethernet storage