Fibre Channel is a high-speed networking technology primarily used for transmitting data among data centers, computer servers, switches and storage at data rates of up to 128 Gbps. It was developed to overcome the shortcomings of the Small Computer System Interface (SCSI) and High-Performance Parallel Interface (HIPPI) by filling the need for a reliable and scalable high-throughput and low-latency protocol and interface. Fibre Channel is especially suited for connecting servers to shared storage devices and interconnecting storage controllers and drives. The Fibre Channel interface was created for storage area networks (SANs).
Fibre Channel devices can be as far as 10 kilometers (approximately six miles) apart if multimodal optical fiber is used as the physical medium. Optical fiber is not required for shorter distances. Fibre Channel also works using coaxial cable and ordinary telephone twisted pair. When using copper cabling, however, it is recommended distances do not exceed 100 feet.Content Continues Below
Fibre Channel offers point-to-point, switched and loop interfaces to deliver lossless, in-order, raw block data. Because Fibre Channel today is many times faster than SCSI, it has replaced that technology as the transmission interface between servers and clustered storage devices. Fibre Channel networks can transport SCSI commands and information units using the Fibre Channel Protocol (FCP), however. It is designed to not just interoperate with SCSI but with the Internet Protocol (IP) and other protocols.
Fibre Channel is also an option -- along with remote direct memory access (RDMA) over Ethernet and InfiniBand -- mostly used in high-performance computing environments for transporting data under the nonvolatile memory express over Fabrics (NVMe-oF) specification to improve flash storage performance over a network. The 100+ member nonprofit technology organization NVM Express Inc. developed NVMe-oF and published version 1.0 of the specification on June 5, 2016. The T11 committee of the International Committee for Information Technology Standards (INCITS) has set forth a frame format and mapping protocol for applying NVMe-oF to Fibre Channel. It finalized and submitted the first version of the mapping protocol, under the FC-NVMe standard banner, to INCITS in August 2017.
Standards for Fibre Channel are specified by the Fibre Channel Physical and Signaling standard and American National Standards Institute (ANSI) X3.230-1994, which is also ISO (International Organization for Standardization) 14165-1.
Development of the Fibre Channel protocol started in 1988 as part of the Intelligent Peripheral Interface (IPI) Enhanced Physical Project, with the first draft of the standard completed in 1989. ANSI approved Fibre Channel in 1994. The first serial storage transport to hit gigabit speeds, Fibre Channel performance has consistently doubled every few years for the last 20 years.
Historically, Fibre Channel networking speeds have been labeled in Gbps -- 1 Gbps, 2 Gbps, 4 Gbps, 8 Gbps, 16 Gbps, 32 Gbps, 64 Gbps and 128 Gbps -- representing throughput performance. The naming convention was changed to Gigabit Fibre Channel (GFC) -- 1GFC, 2GFC, 4GFC, 8GFC, etc. -- by the Fibre Channel Industry Association (FCIA). Each Fibre Channel is backward-compatible to at least two previous generations. For example, 8GFC maintains backward compatibility to 4GFC and 2GFC.
With Generation 5 Fibre Channel, called 16GFC, the encoding mechanism changed. Gen 5 performs at a line rate of 15.025 Gbaud with single-lane throughput of 1,600 MBps and bidirectional throughput of 3,200 MBps, according to FCIA's roadmap.
Gen 6 Fibre Channel added features such as N_Port ID virtualization (NPIV), better energy efficiency and forward error correction (FEC) to improve the reliability of Fibre Channel links and to prevent application performance degradation or outages by avoiding data stream errors. It comes in 32GFC and 128GFC flavors. The former is single-lane at a line rate of 28.05 Gbaud with 6,400 MBps throughput; the latter, with parallel functionality, has four lanes (28.5 Gbaud x 4) for 112.2 Gbaud line rate performance and 25,660 MBps throughput.
The FCIA roadmap extends well into the future to 1 Terabit Fibre Channel (1TFC), which is slated to perform at 204,800 MBps and have its T11 specification completed in 2029. Between that and Gen 6 are generations that include single-lane 64GFC (57.8 Gbaud, 12,800 MBps) and four-lane 256GFC (4 x 57.8 Gbaud, 51,200 MBps) with market availability in 2019 or later. The roadmap also lists more advanced 128GFC and 256GFC versions with estimated T11 specification completion dates of 2023 and 2026, respectively, and a 512GFC (2026 for T11, 102,400 MBps) edition. The roadmap does not yet list line rates or market availability for any of these or 1TFC.
Fibre Channel layers
Fibre Channel defines layers of communication similar to, but different from, the Open Systems Interconnection (OSI) model. Like OSI, Fibre Channel splits the process of network communication into layers, or groups, of related functions. OSI includes seven such layers, while Fibre Channel has five layers. IP networks use packets, and Fibre Channel relies on frames to foster communication between nodes.
The five layers of a Fibre Channel frame include the following:
- Upper Layer Protocol Mapping: FC Layer 4
- Common Services Layer: FC Layer 3
- Signaling/Framing Layer: FC Layer 2
- Transmission Layer: FC Layer 1
- Physical Layer: FC Layer 0
Within a Fibre Channel topology, each of the five frame layers works with the one below it and above it to deliver different functions.
Switches. A Fibre Channel switch enables a high availability, low latency, high performance and lossless data transmission in a Fibre Channel fabric. It determines the origin and destination of data packets to send onto their intended destination. As the main components used in a SAN, Fibre Channel switches can interconnect thousands of storage ports and servers. Features in Fibre Channel director-class switches include zoning to block unwanted traffic and encryption.
Host bus adapters (HBAs). Fibre Channel HBAs are cards that connect servers to storage or network devices. An HBA offloads server processing of data storage tasks and improves server performance. When Fibre Channel and Ethernet networks began to converge, HBA vendors developed converged network adapters (CNAs) that combine the functionality of a Fibre Channel HBA with an Ethernet network interface card (NIC).
Ports. Fibre Channel switches and HBAs connect to each other and to servers through ports, which can be physical or virtual. Data in a Fibre Channel fabric node is sent and received through ports that come in an assortment of logical configurations. Fibre Channel switches can include anywhere from fewer than 10 ports to hundreds of ports in a chassis.
Designs and configurations
The Fibre Channel protocol supports three main topologies to link Fibre Channel ports together for devices such as switches and HBAs to connect servers to a network and storage.
Point-to-point. The simplest and most limited Fibre Channel topology connects two devices (ports) directly together, such as linking a host server to direct-attached storage (DAS).
Arbitrated loop. Devices are linked in in a circular, ringlike manner. Each node or device on the ring sends data to the next node and so on. Bandwidth is shared among all devices, and if one device or port fails, all could be interrupted unless a Fibre Channel hub is employed to connect multiple devices and to bypass ports when they fail. The maximum number of devices that can be in an arbitrated loop is 127, although, for practical reasons, the number is limited to far fewer.
Switched fabric. All devices in this topology connect and communicate via switches, which optimizes data paths using the Fabric Shortest Path First (FSPF) routing protocol and enables multiple pairs of ports to interconnect concurrently. Ports do not connect directly but flow through switches. That means, when one port fails, the operation of other ports should remain unaffected. All nodes in the fabric can work simultaneously, increasing efficiency, while redundancy of paths between devices increases availability. Switches can be added to the fabric without taking the network down.
Interconnection types within the switched fabric topology include the following:
- Single-switch topology is the simplest switch topology, where there is just one switch and no interswitch links. It is seldom used because it presents a single point of failure.
- Cascade topology lines switches up and connects them together one after the other in the manner of a queue. Adding an interswitch link to interconnect the first and last switch in the cascade closes the loop to form a switched fabric ring topology.
- Mesh topology is when every switch in the Fibre Channel fabric connects to every other switch.
- Core-edge topology takes a tiered approach to mesh by using higher-performance director switches as core switches. It connects servers to the edge fabric and storage to core switches. These, in turn, are interconnected to facilitate communication between servers and storage.
- Edge-core-edge topology enables storage and server connects to the edge fabric, but core switch communication is used only to connect and scale edge switches. This topology configuration helps extend the flow of SAN traffic across long distances and eases the management of storage and servers when each are at different edges of a Fibre Channel fabric.
Fibre Channel vs. iSCSi SANs
As a Layer 2 switching technology, hardware handles the entire protocol in Fibre Channel fabrics. By contrast, internet SCSI (iSCSI) is a Layer 3 switching technology that runs over Ethernet. Here, software, hardware or both software and hardware can control the protocol. Ethernet-based iSCSI transports SCSI packets over a TCP/IP network. Because iSCSI uses commonplace Ethernet, it doesn't require buying costly and often complex adapters and network cards. This makes iSCSI cheaper and easier to deploy.
A majority of data centers with a high-capacity SAN for mission-critical workloads prefer Fibre Channel networking over iSCSI. That's mostly because Fibre Channel is a proven entity that they know will reliably handle even the most demanding workloads without dropping data packets.
Specialized installation and configuration skills are required to properly get a Fibre Channel SAN up and running. An IT staff can implement an iSCSI SAN on an existing network using common switches and Ethernet NICs. That means, with iSCSI, there is only one network to build and manage, while Fibre Channel requires two networks: a Fibre Channel SAN for storage and an Ethernet network for everything else.
All major storage vendors today offer iSCSI SAN arrays in addition to their Fibre Channel mainstays. Some sell unified, multiprotocol storage platforms with both iSCSI and Fibre Channel.