Fibre Channel vs. IP vs. InfiniBand

An independent analyst provides insight into one of the most pressing issues in the IT storage arena.


Fibre Channel vs. IP Storage vs. InfiniBand. Fierce debate underlies one of the complexities of storage deployment. Do users deploy Fibre Channel storage area networks (SANs) now, believing that Fibre Channel has a long-term future? Or will all storage-host interconnection eventually go to IP? If so, then where does InfiniBand fit into the equation? As a server interconnect only?

We could easily say that, in the future, there will be no dominant interconnection media. In the future, we will see a mix of Fibre Channel, IP, and Infini Band. If so, however, we beg another, even more complex question: For which applications do we use one versus the other two? Do we use Fibre Channel for host bus adapters (HBAs) and controller interfaces, and IP for the switching fabric? If so, do we negate the assumed benefits of InfiniBand?

The debate over interconnects introduces a major uncertainty factor into IT decision-making: Should I deploy now, deploy half-heartedly, or wait? With the current pace of Fibre Channel development, will IP begin to look like a viable alternative just as the pieces fall into place for Fibre Channel? And, at that point, will InfiniBand be close enough on the horizon to cause users to hesitate yet again? A good deal of IT decision-making hangs in the balance.

FC: The plumbing we have now

Before we had Fibre Channel in the open systems storage world, we had SCSI. In fact, we still have SCSI-the protocol-to help get data into and out of storage devices. Fibre Channel-a high-speed serial transport-replaces the SCSI parallel transport mechanism-often referred to in SAN discussions as "the plumbing."

Compared to SCSI as a transport, Fibre Channel "pipes" provide far more flexibility in terms of distance limitations and switching capabilities. Along the way, Fibre Channel preserves the common SCSI controller API. This is an important point to keep in mind when engaging in at least the part of the debate that pits IP against Fibre Channel-both Fibre Channel and at least one proposed IP Storage standard, iSCSI, preserve the SCSI command set intact. This has allowed both drive manufacturers and systems vendors alike to migrate from parallel SCSI to serial Fibre Channel without a major re-engineering effort.

Fibre Channel, rightly or wrongly, has become synonymous with SANs. A number of vendors reported triple-digit compound annual growth rate shipment numbers for Fibre Channel-related products during 2000. Yet, despite that strong growth, issues remain in the minds of users. Lingering FUD (fear, uncertainty, and doubt) gives IP Storage an exploitable opening. IP SANs are now vying for mind and market share.

IP Storage: Same house, new pipes

It sounds deceptively simple. Reaching back to the plumbing metaphor, one replaces the Fibre Channel pipes with IP pipes to interconnect storage devices to servers in a SAN. The advantages that proponents claim for IP Storage are numerous. Like Fibre Channel, IP is switchable. However, unlike Fibre Channel, IP is mature (e.g., free of interoperability issues that were settled years ago), is well-understood by the IT community at large, and has a huge mass of management software and service offerings available to users. In contrast, users are still struggling to build stable Fibre Channel SANs without considerable help from third parties. Knowledge of Fibre Channel technology within the IT community is anything but widespread.

On the surface, it's a compelling argument. Replace a relative unknown with a known quantity, and get a mature infrastructure complete with management facilities almost for free. But just below the surface of IP Storage, things get complex rather quickly.

First, IP does not guarantee delivery of packets from source to target. Yet SCSI requires that data packets not only arrive at the target destination, but also arrive in the exact order they were sent from the source. Therefore, if you want to send SCSI storage packets over an IP network, some method has to be devised that satisfies the guaranteed delivery requirements of the SCSI protocol.

Second, satisfying the requirements of the SCSI protocol and delivering acceptable performance for a broad range of applications and workloads will require that much of the code required be implemented in specialized chipset hardware. This will not be a trivial task, even for the most expert of IP networking vendors. Additionally, like Fibre Channel, a standard method for coding and decoding IP Storage transmissions must be established. No vendor will make a serious commitment to low-cost, high-volume production of the required chipsets until standards have been established.

At this point in time there is no single "best way" to send storage-related I/O traffic over IP links. What exists now are first-generation implementations available from a handful of vendors and a collection of proposals working their way through Internet Engineering Task Force (IETF) committees aimed at establishing standard IP-based storage transmission methods. They include:

FCIP-FCIP has been proposed as a standard way to link Fibre Channel SAN "islands" via existing IP networks. FCIP can also be used to overcome Fibre Channel's present distance limitations, linking SAN islands over distances greater than those that can be supported by Fibre Channel. FCIP uses encapsulation to send Fibre Channel frames over TCP/IP links. Since Fibre Channel already contains the SCSI protocol, this method satisfies the requirements of the SCSI protocol without a major engineering effort. At the sending end of the transmission, all Fibre Channel frames are converted, or "wrapped," as TCP/IP datagrams. At the receiving end, the datagram "wrapper" is discarded and the Fibre Channel frame is presented to the SAN on the receiving end in its original form and in the order that it was sent relative to the other datagrams that comprise the complete I/O operation. TCP/IP acts purely as a conduit. As such, FCIP datagrams can be routed or switched like any other datagram.

iFCP-iFCP is defined as a gateway-to-gateway protocol used to link Fibre Channel devices via TCP/IP links. The objective is to allow existing Fibre Channel arrays and host bus adapters to attach to an IP network using "lightweight" gateways. With iFCP, standard TCP/IP switching and routing infrastructure components replace equivalent Fibre Channel components. iFCP uses TCP for congestion control, error-detection, and recovery. The same encapsulation format used for FCIP will also be used for iFCP.

mFCP-mFCP is similar to iFCP in that existing IP plumbing is used to interconnect Fibre Channel devices within a storage fabric. However, mFCP uses UDP rather than TCP as the transport protocol, placing responsibility for connection reliability and correct packet sequencing with the IP plumbing layer. Compared to iFCP, mFCP gains performance at the expense of reliability.

iSCSI-Compared to FCIP, iFCP, and mFCP, iSCSI is a much deeper implementation of storage I/O traffic transmitted over IP networks. FCIP preserves Fibre Channel SANs and links them with an IP backbone. iFCP and mFCP replace Fibre Channel SAN plumbing with IP, while preserving the ability of Fibre Channel devices to attach to the IP Storage network. iSCSI banishes Fibre Channel completely from the discussion. Host-based applications communicate with networked storage devices over IP. This means that the entire transmission link from host to device is an IP link. What iSCSI does preserve is the SCSI command set by mapping it to IP. iSCSI preserves the SCSI command set intact, while replacing its transmission protocol with IP. There is a considerable amount of code required to perform the mapping function, introducing a latency factor that is potentially intolerable for many applications. There fore, it is assumed that, to overcome the latency issues, specialized iSCSI chip sets that implement the conversion code in hardware will be needed.

iSNS-iSNS has been proposed as a common discovery, naming, and resource management service for all of the IP Storage protocols now under consideration. iSNS uses a client/server construct. iSNS Clients communicate their attribute information to the server and receive notification of fabric topology-related events from the server. iSNS servers respond to iSNS protocol queries/requests and are the final arbiters of state change.

With the exception of iSNS, all of these protocols have one very important thing in common: All serve as transport mechanisms for the SCSI command set. Hosts communicating with storage devices over IP Storage media still speak SCSI.

IP Storage is still very much in its infancy. None of the protocols mentioned have been accepted as a standard by the IETF. Cisco's iSCSI entry galvanized the storage community to attention, however, and with its announcement, the era of IP Storage officially began.

InfiniBand: New house, new pipes

TCP/IP will continue to serve the IT community well as a high-volume, multi-purpose, networking workhorse. However, it trades off utility against efficiency and performance. IP Storage proponents would have us believe that we can easily get around the potential bottleneck that the IP protocol stack imposes on storage- related I/O with clever engineering. But, that's easier said than done. Enter InfiniBand.

InfiniBand's protocol stack was designed from the beginning to be implemented in silicon. Features and options not efficiently implemented in hardware are omitted, made optional, or left to management software developers to implement. In fact, it is possible that InfiniBand's entire protocol-handling capability can be driven into a single die, with management features either in silicon or in a separate 8-bit microcontroller. Make it simple, make it fast, make it small, make it inexpensive, make it mass-producible: These are the mantras.

InfiniBand will start life as a server interconnect and find its early home within the data center. It replaces the bus-based PCI with a high-bandwidth (multiple gigabytes per second) switched-fabric topology, conceptually compatible with Fibre Channel and IP. The physical fabric combines connectors, cables, and switches. Current specifications call for one-, four-, and 12-wide link options, corresponding to 500MBps, 2GBps, and 6GBps bandwidths. InfiniBand links are designed for data- center distances (approximately 17 meters) using copper cabling, or for intra-building or small-campus distances (approximately 100 meters) with fiber-optic links.

InfiniBand uses a channel-based approach, distributing the I/O processing workload along the communications link. Instead of the memory-mapped "load/store" paradigm of PCI, InfiniBand uses a message-passing "send/receive" model. Adapters take on the responsibility for handling transmission protocols, and InfiniBand switches become responsible for making sure packets get where they're supposed to be.

Where InfiniBand goes as a storage interconnection medium is now in the hands of the InfiniBand Trade Associa tion. Compaq, Dell, Hewlett-Packard, IBM, SGI, and Sun are all charter members. But EMC, Intel, and Microsoft are just as heavily involved, indicating that InfiniBand has implications well beyond server interconnection. Nevertheless, we expect that early InfiniBand storage implementations will be in-rack solutions directly attached to storage devices rather than switched fabrics. We also expect InfiniBand-to-Fibre Channel and Infini Band-to-IP bridging solutions to be forthcoming as well.


We are often asked which storage interconnect technology-Fibre Channel, IP, or InfiniBand-will dominate in the years to come. Our answer: Don't count on any one of the three to dominate the IT world for the foreseeable future. Server-to-storage interconnects are tools used to solve business problems, not religions. Users will implement the best tool available at the time to solve the problem at hand. For example, IP is being increasingly seen as a storage interconnect for long-haul data-replication applications such as EMC's SRDF because of its greater distance capabilities compared to Fibre Channel. On the other hand, Fibre Channel will continue to be the SAN fabric of choice for at least the next two years before reliable and cost-effective InfiniBand and IP alternatives reach the market. And even when they do, Fibre Channel fabrics will not be replaced en masse. It is clear that Fibre Channel and IP, for example, will coexist for some time to come (see chart).

It is incorrect to assume that, just because IP is ubiquitous and inexpensive, it will dominate the storage world. While performance benchmarks for IP Storage solutions have been run, they have been done more as proof-of concept exercises rather than reproductions of real-world, data-center situations. We simply don't know yet where the IP Storage holes are. Until we do, all bets are off.

John Webster is a senior analyst with Illuminata (www.illuminata.com), a research and consulting firm in Nashua, NH.

This article was originally published on July 25, 2001