IP storage and Fibre Channel: Competition and co-existence


A review of the FCIP, iFCP, iSCSI, and iSNS IP storage standards and their potential positioning against the incumbent Fibre Channel and dark-horse InfiniBand.

Click here to enlarge image


In both the IT storage professional community and the storage vendor community, the debate continues unabated: Which technology will provide the best underlying infrastructure for future storage networks?: Fibre Channel (the incumbent), IP (the near-term challenger), or InfiniBand (the long-term dark horse). Users are asking: Do I deploy Fibre Channel now, betting that it has a long-term future? Or will all storage-host interconnection eventually go over the ubiquitous IP? If so, then where will InfiniBand fit? As a server-cluster interconnect only?

It's quite possible that, in the future, there will be no dominant interconnection technology for block-level storage networking. IT organizations may be forced into a mix of all three (or more) protocols. If so, that raises a more difficult question: For which applications do you use one technology versus another? Will Fibre Channel thrive only in the data center, while IP ties in departments and workgroups and serves as a bridge between geographically distant storage area networks (SANs)? Should you use Fibre Channel for the switching fabric at the core of the storage network while deploying IP to reach out to the edges? And, what are the benefits of InfiniBand, and when will it be feasible as a storage technology?

More questions: Should you deploy a SAN now, or wait? Will IP storage emerge as a viable alternative to Fibre Channel just as the pieces (e.g., interoperability and prices) fall into place for Fibre Channel? And, at that point, will InfiniBand be close enough on the horizon to cause users to hesitate once again?

Plumbing we have now

Fibre Channel is a high-speed serial transport that replaces the SCSI parallel transport mechanism, although it still uses the SCSI protocol. In SAN discussions, Fibre Channel is sometimes referred to as the "plumbing."

Compared to SCSI as a transport, Fibre Channel "pipes" provide far more flexibility in terms of connectivity distance and switching capabilities. But Fibre Channel preserves the common SCSI controller APIs (application programming interfaces). This is an important point to keep in mind in the context of the Fibre Channel vs. IP debate: Both Fibre Channel and at least one proposed IP storage standard, iSCSI, preserve the SCSI command set intact. This has allowed both drive manufacturers and systems vendors to migrate from parallel SCSI to serial Fibre Channel without major re-engineering efforts.

Right or wrong, Fibre Channel has become synonymous with SANs. Yet issues remain. Lingering doubt about the long-term viability of Fibre Channel (at least outside the data-center environment) gives IP storage an exploitable opening. In 2002, IP SANs will vie for "mind" and market share.

Same house, new pipes

It sounds deceptively simple: Reaching back to the plumbing metaphor, just replace the Fibre Channel pipes with IP pipes to interconnect storage devices to servers in a SAN. The advantages that proponents claim for IP storage are numerous. Like Fibre Channel, IP is switchable. However, unlike Fibre Channel, IP is mature (e.g., free of interoperability issues, which were settled years ago), is well-understood by the IT community, and has a huge mass of network management software and service offerings available to administrators.

In contrast, building stable Fibre Channel SANs without considerable help from third parties is still a struggle for many IT shops. Comprehensive SAN management software applications for Fibre Channel are still in the developmental stage. And, while knowledge of Fibre Channel technology within the IT community is getting better, Fibre Channel expertise is anything but widespread.

On the surface, IP storage presents a compelling argument: Replace a relatively unknown (and expensive) technology with a known (and inexpensive) technology, and gain a mature infrastructure complete with management facilities. But just below the surface of IP storage, things get complex rather quickly.

Click here to enlarge image

Unlike Fibre Channel, IP does not guarantee delivery of packets from source to target. Yet SCSI requires that data packets not only arrive at the target destination, but also arrive in the exact order they were sent from the source. Therefore, if you want to send SCSI storage packets over an IP network, some method has to be devised that satisfies the guaranteed delivery requirements of the SCSI protocol. Satisfying the requirements of the SCSI protocol and delivering acceptable performance that is comparable to Fibre Channel for a broad range of applications and workloads are the challenges for IP storage vendors.

High-performance implementations of iSCSI, for example, require that most of the protocol processing code be implemented in specialized hardware-a process known as TCP/IP off-load. This has not been a trivial task, even for an expert IP networking vendor. Additionally, like Fibre Channel, a standard method for coding and decoding IP storage transmissions must be established-and adhered to-by vendors to avoid the interoperability issues that have plagued the Fibre Channel community.

Presently, there is no single "best way" to send storage-related I/O traffic over IP links. What exists now are first-generation implementations available from a growing number of vendors, and a collection of proposals working their way through Internet Engineering Task Force (IETF) committees that are establishing standard IP-based storage transmission methods. The evolving IP storage standards include FCIP, iFCP, iSCSI, and iSNS.

FCIP-FCIP has been proposed as a standard way to link Fibre Channel SAN "islands" via existing IP networks. FCIP can also be used to overcome Fibre Channel's distance limitations, linking SAN islands over distances greater than those that can be supported by Fibre Channel.

FCIP uses encapsulation to send Fibre Channel frames over TCP/IP links. Since Fibre Channel already contains the SCSI protocol, this method satisfies the requirements of the protocol without a major engineering effort. At the sending end of the transmission, all Fibre Channel frames are converted, or "wrapped," as TCP/IP datagrams. At the receiving end, the datagram "wrapper" is discarded and the Fibre Channel frame is presented to the SAN on the receiving end in its original form and in the order that it was sent relative to the other datagrams that comprise the complete I/O operation. TCP/IP acts purely as a conduit. As such, FCIP datagrams can be routed or switched like any other datagram.

iFCP-iFCP is defined as a gateway-to-gateway protocol used to link Fibre Channel devices via TCP/IP links. The objective is to allow existing Fibre Channel arrays and host bus adapters to attach to an IP network using "lightweight" gateways. With iFCP, standard TCP/IP switching and routing infrastructure components replace equivalent Fibre Channel components. iFCP uses TCP for congestion control, error detection, and recovery. The same encapsulation format used for FCIP will also be used for iFCP. (mFCP, which is similar to the iFCP standard except that it uses UDP as the transport protocol, was combined with the iFCP proposal at the request of the IETF.)

iSCSI-Compared to FCIP, iFCP, and mFCP, iSCSI is a much "deeper" implementation of storage I/O traffic transmitted over IP networks. FCIP preserves Fibre Channel SANs and links them with an IP backbone. iFCP replaces Fibre Channel SAN plumbing with IP, while preserving the ability of Fibre Channel devices to attach to the IP storage network. In contrast, iSCSI eliminates Fibre Channel from the equation. Host-based applications communicate with networked storage devices over IP. This means that the entire transmission link from host to device is an IP link. What iSCSI does preserve is the SCSI command set by mapping it to IP.

iSCSI preserves the SCSI command set intact, while replacing its transmission protocol with IP. However, there is a considerable amount of code required to perform the mapping function, introducing a latency factor that is potentially intolerable for many applications. Therefore, to overcome the latency issues, specialized iSCSI chip sets, sometimes referred to as TCP Off-load Engines (TOEs), that implement the conversion code in hardware will be needed. (For more information on iSCSI, see "iSCSI: Storage networking for the Internet generation," InfoStor, November 2001, p. 38.)

iSNS-iSNS has been proposed as a common discovery, naming, and resource management service for all of the IP storage protocols now under consideration. iSNS uses a client/server construct. iSNS clients communicate their attribute information to the server and receive notification of fabric topology-related events from the server. iSNS servers respond to iSNS protocol queries/requests and are the final arbiters of state change. (For detailed information on iSNS, see "iSNS: Technical overview of discovery in IP SANs," InfoStor, November 2001, p. 42.)

With the exception of iSNS, all of these protocols have one important thing in common: All serve as transport mechanisms for the SCSI command set. Hosts communicating with storage devices over IP storage media still speak SCSI.

Cisco's iSCSI entry in April 2001 galvanized the storage community to IP storage attention. Presently, however, IP storage is still in its infancy. None of the IP storage protocols have been ratified as a standard by the IETF, although approval of all three is anticipated during 2002. But infancy has not stopped some users, particularly those in the educational community, from experimenting with iSCSI.

An increasing number of vendors are demonstrating iSCSI and other IP storage protocols. For example, a group of vendors recently demonstrated a cross-country deployment of a remote-copy application (see "Vendors demo coast-to-coast IP SAN," InfoStor, November 2001, p. 8). Also, the list of vendors with TOEs (both big and small) gets longer with each passing month, although thus far, none of them are shipping to end users.

New house, new pipes

TCP/IP will continue to serve the IT community well as a high-volume, multi-purpose, networking workhorse. However, it trades off utility against efficiency and performance. IP storage proponents argue that they can solve the potential bottleneck that the IP protocol stack imposes on storage-related I/O with clever engineering. But, that's easier said than done. Enter InfiniBand.

InfiniBand's protocol stack was designed from the beginning to be implemented in silicon. Features and options not efficiently implemented in hardware are omitted, made optional, or left to management software developers to implement. In fact, it is possible that InfiniBand's entire protocol-handling capability can be driven into a single die, with management features either in silicon or in a separate 8-bit microcontroller. The mantras are: Make it simple, make it fast, make it small, make it inexpensive, and make it mass-producible.

InfiniBand will start as a server interconnect and find an early home within the data center. It replaces the bus-based PCI with a high-bandwidth (multiple gigabytes per second) switched-fabric topology, conceptually compatible with Fibre Channel and IP. The physical fabric combines connectors, cables, and switches. Current specifications call for one-, four-, and 12-wide link options, corresponding to 500MBps, 2GBps, and 6GBps bandwidths. InfiniBand links are designed for data-center distances (approximately 17 meters) using copper cabling, or for intra-building or small- campus distances (approximately 100 meters) with fiber-optic links.

InfiniBand uses a channel-based approach, distributing the I/O processing workload along the communications link. Instead of the memory-mapped "load/store" paradigm of PCI, InfiniBand uses a message-passing "send/receive" model. Adapters take on the responsibility for handling transmission protocols, and InfiniBand switches will be responsible for making sure packets get where they're supposed to be.

Where InfiniBand goes as a storage interconnection medium is now in the hands of the InfiniBand Trade Association. Compaq, Dell, Hewlett-Packard, IBM, SGI, and Sun are all charter members. But EMC, Intel, and Microsoft are just as heavily involved, indicating that InfiniBand has implications well beyond server interconnection. Nevertheless, early InfiniBand storage implementations will probably be in-rack solutions directly attached to storage devices, rather than switched fabrics. It is also likely that InfiniBand-to-Fibre Channel and InfiniBand-to-IP bridging solutions will be forthcoming as well.


IT managers often ask which storage interconnect technology-Fibre Channel, IP, or InfiniBand-will dominate in the years to come. The answer: Don't count on any one of the three to dominate for the foreseeable future. Server-to-storage interconnects are tools used to solve business problems, and users will implement the best tool available at the time to solve the problem at hand

It is incorrect to assume that, just because IP networking is ubiquitous and relatively inexpensive, that IP storage networking will dominate. IP storage is a new technology with uncharted territory. However, IP storage does have strong support from three 800-pound gorillas-Cisco, IBM, and Intel. None of the three really compete with each other, and their combined strength could spur relative rapid adoption of IP storage in the end-user community.

On the other hand, Fibre Channel will continue to be the SAN fabric of choice for at least the next two years or until reliable and cost-effective IP and InfiniBand alternatives reach the market. And even when they do, Fibre Channel fabrics will not be replaced en masse. It is clear that Fibre Channel and IP, for example, will co-exist for some time to come.

InfiniBand as a storage networking interconnect technology remains a contentious dark horse. However, some believe that InfiniBand is really a Trojan horse: Once in the data center as a server-to-server interconnect, users will naturally extend it outside the server environment to external InfiniBand storage fabrics. Yet, the InfiniBand forecast from storage vendors is cloudy, at best. Systems vendors that also have storage divisions are the more obvious proponents of InfiniBand storage, while independent storage networking vendors maintain a façade of agnosticism: "We will go where our users go." The first phase of the InfiniBand rollout will be in server-only configurations. The next phase will begin to include data-center storage. The emergence of bridging products that connect InfiniBand server area networks to Fibre Channel SANs could appear as early as the third quarter of 2002.

The reality of IT is that enterprise users adopt standards. Whether the standards are commissioned or de facto, when projects are planned and purchase orders cut, the decisions that stand one interconnect technology up against another have to be made. Typically, those decisions are aligned with an internal policy to standardize around as few alternatives as possible to reduce complexity. Therefore, there is no question that each technology-Fibre Channel, IP, and InfiniBand-will compete for a share of the overall storage interconnect market. The opportunity available to storage networking vendors is bounded by IT storage budgets that, in uncertain economic times, will receive intense scrutiny. The competition among the leading proponents of each of the three technologies will be equally intense in 2002.

John Webster is a senior analyst with Illuminata (www.illuminata.com), a research and consulting firm, in Nashua, NH.

This article was originally published on December 01, 2001