Fibre Channel over Ethernet (FCoE) is a new protocol that expands Fibre Channel into the Ethernet environment. FCoE combines, and leverages the advantages of, two technologies: the Fibre Channel protocol and Ethernet.

The terms unified fabric or I/O consolidation are synonymous and refer to the ability of a network (in this case, Ethernet) to use the same switches and host adapters to carry different types of traffic that have very different traffic characteristics and handling requirements.This equates to being able to install and operate a single network instead of two or three.

With FCoE, converged network adapters (CNAs) replace and consolidate network interface cards (NICs), host bus adapters (HBAs), and host channel adapters (HCAs). This requires a lower number of PCI slots on rack-mounted servers, and it is particularly beneficial in the case of blade servers where often a single mezzanine card is supported per blade.

Data center bridging

The biggest challenge of I/O consolidation is to satisfy the requirements of different traffic classes with a single network without creating “traffic interference” (i.e., without having one class of traffic starving another). For this reason, the IEEE 802 standards group has created a working group named Data Center Bridging (DCB) to extend the Ethernet standards to satisfy these new data center requirements.

DCB activities are carried out under the names of:

  • IEEE 802.1Qbb (Priority Flow Control)—The basic mechanism to enable a “lossless Ethernet” per priority (i.e., Ethernet does not lose frames under congestion, which is required for FCoE);
  • IEEE 802.1Qau—A Layer 2 congestion management protocol;
  • IEEE 802.1Qaz—A standard that covers bandwidth management and the Data Center Bridging Exchange (DCBE) protocol (the control protocol used to negotiate extensions to Ethernet).

The figure “Layering over extended Ethernet” shows that this extended Ethernet (also called Converged Enhanced Ethernet, or CEE, or Data Center Ethernet, or DCE) platform can support all the current protocols, plus FCoE, Remote DMA (RDMA), and Inter Processor Communication (IPC).

Standards update

All Data Center Bridging standards are in an advanced phase of standardization and are expected to reach the standard status level in the first half of this year.

FCoE is standardized in the FC-BB-5 Working Group of the INCITS T11 Technical Committee—the standards body that deals with Fibre Channel. The draft standard went in T11 letter ballot in October 2008, and the final standard is expected to be finalized by June after completion of the letter ballot comment resolution process.

Pragmatically, there is an industry-wide agreement on these technologies, and some companies have introduced (or are planning to introduce) DCB and FCoE products.

FCoE and I/O consolidation

FCoE is based on the observation that Fibre Channel (FC) is the dominant storage protocol in the data center, and that any viable I/O consolidation solution for storage should be based on the FC model. The ideas behind FCoE are simple:

  • To extend Ethernet with FC-like features to support lossless transmission;
  • To carry each FC frame intact inside an Ethernet frame.

The encapsulation is a frame-by-frame encapsulation and, therefore, the FCoE layer is completely stateless and does not require fragmentation and reassembly.

The figure “I/O consolidation” shows an example of I/O consolidation with FCoE, in which the only connectivity needed on the server is Ethernet, while separate backbones can still be maintained for LANs and SANs.

Seamless integration

This is easy to implement, since FCoE is part of the Fibre Channel architecture; that is, it guarantees seamless integration with existing FC SANs, and reuse of existing FC SAN tools and management constructs. Another important advantage is that FCoE does not require gateways. In fact, the encapsulation/de-encapsulation functions simply add or remove an Ethernet envelope around an FC frame: the FC frame is untouched and the operation is completely stateless.

For example, zoning is a basic provisioning function that is used to give hosts access to storage. FCoE switches continue to offer an unmodified zoning function, ensuring that storage allocation and security mechanisms are unaffected. The same consideration applies to all other Fibre Channel services, such as dNS, RSCN and FSPF.

The FCoE encapsulation is shown in the figure “FCoE frame format.” Starting from the inside out, there is the FC frame that can be up to 2KB; hence, the requirement to support Ethernet jumbo frames (up to 2.5KB). The FC frame contains the original FC-CRC. This is extremely important, since the FC frame and its CRC go end-to-end unmodified, independently of being carried over FC or FCoE.

Next is the FCoE header and trailer that mainly contain the encoded FC start of frame and end of frame (in native FC, these are ordered sets that contain code violation and, therefore, they need to be re-encoded). Finally, the Ethernet header contains Ethertype = FCoE and the Ethernet trailer that contains the Frame Control Sequence (FCS). The reserved bits have been inserted so that, even in the presence of minimum-size FC frames (28 bytes), the payload is greater or equal to the minimum Ethernet payload (46 bytes).

In blade servers, intermediate Ethernet switches between the CNAs and the FCoE switches may be present inside the blade server chassis. They are not fully functioning FC switches; they just snoop the FCoE Initializations Protocol (FIP) to better support FCoE (for example, by installing security ACLs to protect FCoE operations).

Another way of looking at FCoE is that it moves FC over the physical layer of Ethernet, eliminating the need to develop a new, faster, physical standard for FC. The volumes associated with Ethernet make this an interesting economical choice. It also provides a future roadmap for FC at 40 and 100Gbps as Ethernet moves toward these standards.

How FCoE differs from iSCSI

As shown in the figure “Block protocol summary,” the FCoE layer replaces the TCP/IP layer used in iSCSI, and relies on the DCB improvements in Ethernet.

Since FCoE was designed as a data center technology with no IP layer, it is not routable over IP; however, FCoE routing will be performed using already established protocols, such as FCIP. iSCSI can be implemented in ‘lossy’ networks, whereas FCoE requires a ‘lossless’ network. The table below summarizes some of the differences between FCoE and iSCSI.

iSCSI deployments to date have typi- cally been in Tier 2 and 3 data centers in large organizations, in the core data center of small/medium enterprises, and in remote offices. The application sweet spot has been storage consolidation for business-critical Windows environments, and the majority of installations run on Gigabit Ethernet.

Fibre Channel deployments dominate Tier 1 data centers in large organizations, and typically run mission-critical applications in UNIX and Windows environments. Examples of common workloads include data warehousing, data mining, enterprise resource planning, and OLTP. The majority of installations run on 4Gbps FC.

FCoE deployment scenarios

FCoE expands Fibre Channel into the Ethernet environment and is an evolutionary approach to I/O consolidation. The following five scenarios examine where FCoE is likely, or not likely, to be deployed.

Scenario 1: Existing data centers with large investment in FC storage
Most large data centers have a separate dedicated storage network using Fibre Channel. FC is the SAN fabric of choice for Tier 1 data centers. Data centers with large FC installations are likely to con- sider FCoE networks to leverage and extend investments in their FC SAN and management infrastructure. Transitions to FCoE will likely begin at the edge, where new servers and edge switches will be updated to support FCoE and DCB enhancements, providing access to existing data on FC storage, as shown in the figure “Transition from FC to FCoE.”

Scenario 2: New data center installation
With a new data center, the choice is less clear because FC, FCoE or iSCSI may satisfy the required I/O load. Assuming the company is planning an Ethernet infrastructure, then installation timing and existing IT personnel skill sets become the deciding factors. If the data center is coming on line now or in the near future, then iSCSI and FC are viable options; however, if the data center is planned to come online in six months or more, then FCoE may be the more viable option.

Companies with Fibre Channel SAN expertise will most likely consider FCoE, since it leverages existing staff and expertise; however, companies with limited or no Fibre Channel experience are likely to leverage their Ethernet and TCP/IP expertise and opt for iSCSI or FCoE.

Scenario 3: Existing data centers with a mix of storage (DAS, FC, NAS) requiring near-term data center consolidation
Data centers today may be looking for consolidation solutions for any of the following reasons:

  • Reduce cost through server virtualization;
  • Reduce environmental costs (power and cooling);
  • Improve manageability (reduced labor expenses);
  • Improve service levels for customers;
  • Space constraints.

The decision criteria usually revolve around existing storage provisioning and data management practices. In existing NAS- dominated networks, IT is likely to continue with iSCSI. In existing Fibre Channel networks, FCoE is likely to be deployed, since it leverages existing SAN management and infrastructure. It is also likely that the environments will remain a mixture of protocols, based on which is best suited for a particular application.

Scenario 4: Small to medium data centers
For small to medium-size businesses, the choice between FCoE and iSCSI largely depends on budget, IT personnel skill set, and application requirements. iSCSI storage networks will continue to be successful and satisfy the performance requirements for the majority of business applications in this segment, while also providing good value. Where high-availability or performance-oriented databases are the requirement, both FCoE and iSCSI are viable options.

Scenario 5: Remote office installations
For remote offices, iSCSI offers the benefits of native IP routing to address longer distances than FCoE, which requires some form of Ethernet bridging. Remote offices will likely remain mostly iSCSI based.

What will happen to FC?

Today, 8Gbps Fibre Channel is available from a variety of HBA and switch vendors, and disk array vendors will support 8Gbps systems later this year. And 16Gbps FC is planned for market availability in 2011. According to IDC, the FC SAN market is expected to reach $11 billion in 2009. Fibre Channel will be around for a long time, and existing customers can either migrate to 8Gbps and 16Gbps FC or move to FCoE.

In either case, Fibre Channel customers have a future roadmap, and their investment is protected.

FCoE extends, rather than replaces, Fibre Channel, allowing organizations to seam- lessly integrate their Ethernet and Fibre Channel networks at the pace and on the path that work best for them. Combined with enhancements to Ethernet, FCoE gives data centers the ability to consolidate their server I/O and simplify network infrastructure, saving both capital and operational expenses while increasing flexibility and control.