Problem solve Get help with specific problems with your technologies, process and projects.

FCoE SAN multi-hop technology primer

FCoE SAN is growing in popularity, but the lack of multi-hop switching support remains a problem. Find out what vendors and users are doing to improve FCoE performance.

What you will learn in this tip: Fibre Channel over Ethernet (FCoE) storage-area network (SAN) technology is becoming more popular in data storage environments, but there are performance issues, primarily the lack of multi-hop switching support, that need to be addressed that could potentially stunt the growth of the technology. Find out what vendors and users are doing to improve FCoE SAN performance.

FCoE SAN is gaining broad support from storage and network vendors, and customer adoption is also rising. Because it's a new protocol and relies on many new features, FCoE remains somewhat limited in terms of interoperability and flexibility. One often-criticized element is the lack of multi-hop switching support in FCoE SANs, but what exactly does this mean?

A quick Fibre Channel primer

Fibre Channel (FC) initiators contain a number of Node Ports (“N_Port”) that connect to the Fabric Ports (“F_Port”) on switches. FC switches talk to each other using Expansion Ports (“E_Port”) before finally communicating with the N_Port on the storage array. This allows them to route traffic through the SAN to avoid data loss and congestion. FCoE SANs adopt a virtual version of this configuration, with a “VN_Port” talking to a “VF_Port,” and (if they support it) the network switches using “VE_Ports” to exchange data over an inter-switch link (ISL).

One major difference between a Fibre Channel fabric and Ethernet network is intelligence: The fabric itself actively participates in access and routing decisions. Although it's distributed, the FC fabric has some intelligence and thus FC switches are more involved in the network than basic Ethernet switches. In particular, each switch participates in making decisions about where to send data traffic, so each stream of initiator-to-target traffic gets its own route through the SAN rather than sharing a single route as in an Ethernet LAN with spanning tree path management.

FCoE allows various Ethernet switches to be part of the SAN. Many assume the FCoE ideal is for each Ethernet switch to be a full FC Forwarder with ISLs, and thus an active participant in the SAN just like a traditional FC switch. But it's possible and sometimes desirable to skip some or all of the SAN processing, and this causes much confusion in the press and among end users. A given Fibre Channel over Ethernet SAN might have a simple Ethernet edge for connectivity, a multi-tiered Ethernet switch infrastructure or be a full end-to-end fabric, depending on which features are used.

Edge-only FCoE

The earliest implementations of FCoE restricted Ethernet to the edge facing the hosts. This allowed FCoE to function in the absence of many data center bridging (DCB) technologies, and gave Cisco Systems Inc. and EMC Corp. bragging rights for being early implementers of the new storage protocol.

Edge-only FCoE relies on a full FC SAN, including storage arrays, switches and fabric services, but allows hosts (or initiators in storage lingo) to connect using 10 Gb Ethernet (10 GbE) rather than FC HBAs. This Ethernet edge provides some cost and flexibility advantages, though these were modest at the start. But, more importantly, edge-only FCoE allows the protocol to see some real-world deployment and experimentation.

But edge-only FCoE doesn't deliver on the “unified networking” promises made by proponents. At best, it's a transition technology for companies wishing to leverage a large existing Fibre Channel investment and vendors that don't yet support advanced FCoE and DCB protocols. Edge-only FCoE will be phased out along with the base FC protocol, with multitiered and multi-hop FCoE SANs evolving to replace them now.

A natural multi-hop FCoE SAN evolution?

Although not the only approach for Fibre Channel over Ethernet, most would expect the final evolution of the protocol to be Ethernet switches with full FC fabric capabilities. This includes fabric login and ACL enforcement, VE_Port ISLs and A/B fabric isolation. Such an FCoE switch is truly a complete FC switch, and maintains the assumptions and best practices employed in the Fibre Channel world today.

But this may not be the ideal situation. “Domain ID sprawl” might force the use of NPV even for fully capable FCoE switches, and many sites won't want the learning curve and operational load that comes with full FCoE SANs. Also, this political aspect might kill full FCoE before it starts: Network administrators might push back on building out Fibre Channel over Ethernet fabrics, and iSCSI or NFS might halt the growth of the protocol. Ultimately, it’s possible that the demise of FC might also fatally stunt the growth of FCoE.

Three FCoE alternatives

For now, there are three different approaches taken by various vendors and architects to improve the integration of Fibre Channel over Ethernet:

1. Ethernet-only bridging: As long as all of the Ethernet switches include the DCB extensions required to deliver FCoE traffic correctly, they don’t need to participate in the SAN at all. They can simply pass Ethernet frames as always, relying on the edge switches to integrate with the Fibre Channel endpoints. Such “non-participating” switches can't pass traffic along multiple routes without extra functionality, such as Brocade Communications Systems Inc.’s VCS Ethernet fabric technology.

2. FIP snooping: One technique to improve the integration of FC SANs is to allow the Ethernet switches to “snoop” on the FCoE traffic. These switches can then enforce FC settings and assist in fabric login without being full participants in the FCoE fabric. These devices exist in a grey area, with some architects calling such SANs “multi-hop” and others preferring the term “multitier” to differentiate them from full Fibre Channel or Fibre Channel over Ethernet fabrics.

3. NPV switches: One issue with large Fibre Channel SANs is the proliferation of so-called “domain IDs” and resulting limits in scalability. Both of the alternatives listed above avoid this situation simply by not participating in the SAN fabric, but there is another alternative: NPV switches virtualize the devices “behind” them, similar to how NPIV or NAT enable one endpoint to serve multiple devices. This technique is also popular for FC switches, and such SANs can rightly be called “multi-hop” even without ISLs and VE_Ports.

Each of these alternatives is a valid approach to building a multi-hop FCoE SAN when used appropriately: Each allows multi-switch scalability beyond the limited “edge-only” approach that was the hallmark of first-generation FCoE. But none matches the textbook multi-hop FCoE SAN envisioned by those familiar with existing FC technologies.

BIO: Stephen Foskett is an independent consultant and author specializing in enterprise storage and cloud computing. He is responsible for Gestalt IT, a community of independent IT thought leaders, and organizes their Tech Field Day events. He can be found online at, and on Twitter at @SFoskett.

Dig Deeper on Ethernet storage