Two protocols—iFCP and FCIP—tap the scalability of the Internet Protocol and Ethernet networks, and can be used to overcome Fibre Channel networking limitations when connecting SAN islands. But iFCP and FCIP operate very differently; find out which one is right for your environment.

What switches support which protocol
Brocade Communications Systems Inc.: Now supports FCIP, but had relied on Computer Network Technology (CNT) Corp. partnerships to provide distance solutions.
Cisco Systems Inc.: Heavily invested in FCIP with a rich but proprietary feature set.
CNT: Supports FCIP. McData may support both protocols after its acquisition of the firm.
McData: iFCP after buying Nishan Systems.
Growth is one of the most difficult challenges facing storage managers with Fibre Channel (FC) SANs because the FC protocol, by design, sacrificed scalability for speed. The Internet Protocol (IP) and the Ethernet network offer scalability that two variations of the FC protocol--iFCP and FCIP--use to connect SAN islands and overcome FC networking limitations. Unfortunately, each switch vendor supports only a proprietary version of iFCP or FCIP (see What switches support which protocol, this page). Storage managers need to understand the high-level differences between these two protocols to make better decisions on which protocol/switch best fits their SAN infrastructure and expansion needs.

Typically, SAN growth problems arise in one of two ways. Storage managers with small- and medium-sized SANs tend to create SAN islands to overcome scaling issues, but continued growth increases management complexity. Storage managers with large SANs have successfully implemented director-class switches, but they must now connect their large SAN to a secondary data center SAN at a remote site because of new disaster recovery requirements. Each of these growth challenges can be solved by connecting separate FC SANs over a TCP/IP network.

At a basic level, the iFCP and FCIP protocols connect separate FC SANs by encapsulating FC frames within TCP packets and using the LAN/WAN infrastructure as the transport layer. A storage manager should expect that either protocol will:

  • Scale to connect multiple SAN islands or multiple data centers.
  • Provide resiliency and recovery from LAN/WAN failures.

But the Internet--and IP--don't expect the same level of component reliability that is taken for granted in FC SANs. A server can appear and disappear on an Ethernet network using IP without disruption unless it was actively receiving or transmitting data; even then, only the clients at the other side of the data flows are likely to notice. The loss of a node on an FC SAN, however, will result in state change notifications because the FC SAN assumes longevity in connection states. For this reason, resiliency and recovery are important to the storage manager because while the LAN/WAN may be stateless, FC SANs aren't.

Pros and cons of FCIP and iFCP


The FCIP protocol
At a very high level, FCIP is a tunneling protocol that creates a logical tunnel to transport the FC packet from one FC SAN to another over IP. An FCIP gateway switch is installed on each SAN. Each FCIP port contains one or more FCIP tunnels. Each tunnel is a virtual connection point, and each tunnel is given an IP address. When the FCIP switch receives an FC packet, the packet is wrapped as data into an IP packet and sent along. It's analogous to putting an FC-addressed envelope into an IP-addressed envelope and giving it to the IP postman to deliver. There's some overhead because of the larger IP envelope that's needed, but it's a relatively simple process--although there might be special delivery charges for the use of the tunnel if it has to be specially provisioned.

FCIP is most often implemented to solve primary-to-secondary data center disk replication; unfortunately, it doesn't scale well beyond this application. This is because the tunneling design creates an interconnection between two FC SANs that makes them, logically, a single FC SAN, but leads to a proliferation of tunnels. If, for example, a user wishes to connect an Atlanta FC SAN with a Boston FC SAN, then only an Atlanta/Boston (AB) tunnel is needed. If the Chicago FC SAN has to be brought online to share data with Atlanta and Boston, an Atlanta/Chicago (AC) and a Boston/Chicago (BC) tunnel will be needed, making three data centers and three tunnels. If you add Denver to the mix, three more tunnels will be required for a total of four data centers and six tunnels, but it would still be one logical SAN. The geometric growth of these tunnels occurs because the network is essentially flat; reducing the number of tunnels would result in more hops across the latency of IP WANs.

Even when this tunneling growth can be managed easily, there's another limitation that hampers SAN growth through FCIP. Because all connected FC SANs become a single logical FC SAN, all of the FC protocol limitations apply, including the limitation of 239 switches in a fabric and a single fabric name server to manage it all. Any topology changes at any of the connected sites are transmitted to the other SANs. This is why fundamental changes in the FC protocol itself will be required to make FCIP scalable. Another approach to this limitation is to add proprietary methods of segmentation at the tunnel points. This approach, however, won't allow a mix of vendor connections on any single tunnel.

The resiliency of FCIP is limited to the resiliency of the IP tunnel. Storage administrators have to be aware of this limitation because the FCIP protocol can't easily recover from LAN/WAN connection losses without administrator intervention. When a problem occurs in the LAN/WAN, the tunnel may be lost. If no other path exists, the fabric will be split into two fabrics that will each undergo a fabric reconfiguration with the usual outage potential. When the tunnel is recreated, a second outage, managed by a storage administrator, will occur as one of the fabrics is brought down and then back up to re-establish the single logical fabric configuration.

Vendor interoperability won't become a feature of the FCIP protocol until the Internet Engineering Task Force (IETF) and/or T11 (a group within the InterNational Committee for Information Technology Standards) ratifies FCIP as a standard that would allow compliant switches from different vendors to interconnect. Ratification is further complicated because to make FCIP work well, vendors will have to give the existing FC protocol some means of recognizing the latencies and scaling issues IP connections introduce. But until the standards committees provide FCIP standardization, there won't be an obvious market for independent software vendors that might create new solutions to improve the IP WAN link transfer. In the absence of proven standardization, storage managers should plan on using a single FCIP vendor's switch at each side of the tunnel. This will add complexity if one of the connected SANs is composed of another vendor's switches. In that case, proof-of-concept testing should focus on the inter-switch link between the two vendors' switches.

At the core of the development problems for the FCIP protocol is that its design emulates rather than implements TCP/IP routing methodologies. A similar design approach was used for FC. The FC protocol has many network components whose designs were based on an IP-like network, but the designs were modified to favor speed and accuracy. For instance, the limitation of 239 switches on a single fabric came about because it allowed all routing to be based on a single leading byte in the header. Because only one byte of the entire packet had to be read to make a path choice, it allowed for incredible speed. Zoning within FC is somewhat analogous to an IP virtual LAN. The design goal is to isolate one group's traffic from another's at a layer above the routing layer.

There's an inherent weakness in this FCIP approach to mimicking IP solutions. Vendors are currently implementing proprietary FCIP mimics of the IP features that the market needs. The accumulation of non-standard features will tighten the lock on a vendor's customers because of the cost of replacing the incumbent vendor's infrastructure.

Best suited for...
FCIP is best suited for:
Connecting primary and secondary SANs over a WAN for disaster recovery objectives.
iFCP is best suited for:
Connecting SAN islands within a single data center while preserving fabric segmentation.
SAN implementations that have remote sites that may appear, disappear and reappear according to business needs.

Inside iFCP
In contrast to FCIP, iFCP translates 24-bit FC addresses to the Internet addresses of the IP protocol. The data in the FC packet is roughly equivalent to the data in the IP packet but, to use the envelope analogy again, the FC-addressed envelope is given an IP-addressed label before it's given to the IP postman to deliver. There's no overhead, but the labeler has to have enough intelligence to write the new label quickly. When the envelope is opened, the data is revealed, rather than another envelope as with FCIP.

Because of the translation service between FC and IP, an iFCP fabric on one side of an iFCP port sees all of the fabrics on the other side of the IP link as separate and autonomous fabrics, each with an IP address. Using our earlier example, this means the Atlanta SAN would need only a single IP access point to send out all of its IP packets that are routed to Boston, Chicago or wherever. Adding a new data center doesn't require new tunnel provisioning for all associated pre-existing data center switches, although the switch would have to be made aware of the new IP/FC association.

The translation provides an easy segmentation point, and a fabric on one side of an iFCP connection remains autonomous from the fabric at the other side with all of its own fabric-related services. Each fabric is still subject to the FC protocol limitation of 239 switches, but this limitation is confined to each connected SAN rather than to the overall logical SAN that includes all of the connected SANs. A single but very large data center SAN, or even a campus SAN, can be segmented in the same way that SAN islands across a WAN would be segmented. Within a large FC SAN, the FC e-port connections between switches could be replaced with iFCP connections. In this way, a department might have a SAN with only an occasional need to connect to a larger local backup SAN. The iFCP connections become a means of reducing the effect of a fabric reconfiguration to functionally isolated switches.

The resiliency of iFCP is identical to that of the Internet because iFCP requires only ping-able access with a low drop rate from one iFCP access point to another to function properly. In practice, storage administrators are very unlikely to drop their iFCP packet out onto the public Ethernet or to simply trust the speeds available on their company's private intranet connections. Both will require careful networking provisioning and detailed knowledge of network traffic pattern ebbs and flows. But iFCP provisioning is easier than FCIP provisioning because the latter adds the complexity of tunnel management to the management of private intranet connections.

iFCP is more resilient than FCIP because autonomous fabrics can better accommodate opportunistic connections between remote FC SAN islands. The impact of a connection loss to a fabric can be tolerated by the infrastructure until such time as a data flow is needed.

The innovative potential for iFCP far exceeds FCIP because iFCP has the means of representing all end nodes of the FC SAN with separate and unique IP addresses. The network address translation from FC WANs to IP addresses lets IP vendors apply mature IP solutions such as time-of-day allocations down to the storage port on a particular disk frame or an FC tape drive in a library. This potential will always be far beyond the reach of the FCIP switch's vendor development teams, who will need to write the solution into FC-based tools before the user can reap any benefit.

Some users may anticipate the demise of FC as iSCSI gains popularity; indeed, high-priced FC might be dealt a fatal blow when enterprise-class iSCSI disk frames arrive. Many managers will have to decide whether to pay high FC SAN prices now or wait for wider adoption of the iSCSI solutions.

iSCSI networking will always remain separate from FCIP networking, and an FC hardware vendor will need to provide the bridge. The iFCP protocol design creates the bridge that can allow FC-based disk frames to communicate with iSCSI server host bus adapters because it already incorporates a Network Address Translation between FC and IP.

Users tend to dismiss the implications of protocol choices as details for vendors to work out. But vendors are working out the details of their own solutions, creating features to meet the current business needs their marketing departments have identified. The FCIP protocol will prolong the current reliance on FC networking. The iFCP protocol will shift this reliance to TCP/IP networking solutions, the solutions in place for iSCSI implementations. Ultimately, iFCP and iSCSI SANs may appear indistinguishable to administrators.

Dig Deeper on SAN technology and arrays