Fibre Channel vs. Ethernet/iSCSI: Déjà vu

If history repeats itself, it'll be déjà vu all over again.


One of the hottest debates in the storage industry is raging about whether Fibre Channel will survive against IP/Ethernet storage and the emerging iSCSI standard. But many of the debaters seem unaware that we've been here before. In fact, it's possible to draw parallels to situations in the past and use those to predict the winners and losers. This can provide insight into why certain approaches are likely to be winners or losers.

First of all, the solution that is technically "better" or "more elegant" often loses. That's not to say that superior technology is not important, but that technology alone cannot win. Superior economics trumps technological elegance, and the market has a strong track record of favoring easy, cost-effective architectures that are "good enough."

The old IT maxim, "Fast, good, cheap: Pick two," is missing a critical dimension. (It's also misleading in implying that each of the three has a similar impact on technology decisions.) A more realistic model for evaluating the likelihood of technologies succeeding on a large scale looks more like this:

AttributeWeighting factor

This may not be the way these weights should look, but in the real world it's pretty close to the way IT managers value these attributes: History shows that cheaper and easier nearly always win. This indicates that given the ability of several competing technologies to perform the job at hand, selecting technology winners is not simply (or even primarily) a technical question. Economics and the ability to play in the real world are in fact the primary determinants of technology acceptance. This, in turn, is driven by volume economics and design simplicity.

For example, Ethernet was not always the preferred LAN medium. Competing technologies such as Token Ring and Fiber Distributed Data Interface (FDDI) bowed to the unstoppable juggernaut of Ethernet because of one critical factor: Ethernet wins because it's "dumb," and dumb is cheap. "Dumb" in this case means that Ethernet, in full alignment with the philosophy that underlies the Internet, puts as little intelligence as possible into the network itself and pushes the intelligence to the hosts, where changes to that intelligence are inevitably cheaper and easier. This results in very simple and inexpensive network host adapters as well as devices such as routers and switches.

IP succeeded over the "more elegant" OSI network protocols for the same reason: OSI was complex and hard to implement, making it expensive and difficult, while IP was inexpensive and easy. Note that such simplicity pays double as it hits both of the high-weight items in the table, reducing cost as well as making the technology fundamentally easier to use and deploy.

This model can be applied to storage networking as well as other technologies. The upcoming battle in the storage area networking space will be largely between Fibre Channel and iSCSI/Ethernet (and possibly InfiniBand). In each case, the storage protocol is SCSI; the differences are in the physical networks and related network switching gear, and there are also some significant differences at the protocol level in terms of how SCSI is carried. The simplified methodology of reviewing each technology in light of the four key attributes in the table on the left provides a good tool to predict how this discontinuity will impact storage networking.


Both Fibre Channel and iSCSI/Ethernet have significant strengths in different areas. From a purely technical perspective, Fibre Channel offers guaranteed delivery and immunity from congestion at the lower layers of the protocol stack. In today's world of dedicated full-duplex switched Ethernet connections, though, these advantages are increasingly academic. In any case, TCP also provides guaranteed in-order delivery (at a higher level of the stack), and switches eliminate congestion. Remember that Token Ring and FDDI also offered these same advantages of Fibre Channel, but both have fallen to Ethernet.

The iSCSI/Ethernet approach may also have an edge in interoperability. Although there have been significant strides in Fibre Channel interoperability recently, many storage area network (SAN) users can attest that Fibre Channel implementations involving multiple vendors may not be fully interoperable, especially when mixing switches. Ethernet and IP, on the other hand, are the "gold standard" of interoperability: It's hard to find a product combination that won't work. What effect iSCSI will have remains to be seen, but it has one very important thing going for it: iSCSI is being defined through the Internet Engineering Task Force (IETF) standards process, which has proven its ability to generate interoperable implementations. Industry organizations such as the Fibre Channel Industry Association may be able to guide interoperability, but the IETF methodology has a far better track record, partly because it involves a more diverse crowd early in the process.

Furthermore, although Fibre Channel defines a handful of useful profiles (including IP, to be fair) at its top layer, there are IP encapsulations for nearly any protocol of interest, allowing IP to offer far superior flexibility in the future. IP-based solutions appear at this point to be the ultimate in futureproof technology, as evidenced by the progression of Internet technology. That said, there are clearly areas where Fibre Channel can play today that Ethernet/IP cannot, in particular in the FC-AL role of directly attaching to storage devices such as disk and tape drives.

Fibre Channel does have advantages in terms of some of its more-advanced features such as Quality of Service (QoS) through bandwidth guarantees, and the impressive but somewhat underutilized FC-3 services like hunt groups, striping, and multicasting. Although QoS is available but not yet widely used in the IP world, features like hunt groups may be a long time in coming.

Finally, although we can thank Fibre Channel for freeing us from the distance limits and hideous cabling and termination problems of SCSI hardware, the distances are still limited to a few kilometers. The use of iSCSI provides complete geographic abstraction, making it possible to access storage resources anywhere there is an IP connection, although performance will clearly be a function of latency and bandwidth. This capability is something new and different and may well lead to a whole new class of remote replication and synchronization capabilities leveraging the IP network infrastructure.


"Faster" is the attribute that garners a disproportionate amount of interest and attention. Speed and performance are governed by both hardware and software factors. The biggest difference here is one of philosophy: Is it better to reinvent the wheel, or ride the wave? The markets teach us that sub-optimal solutions can be better, especially when they effectively leverage existing infrastructure and knowledge bases.

On the hardware side, Fibre Channel has stiff competition from Ethernet. Currently, Fibre Channel is available in 1Gbps and, with some devices, 2Gbps speeds. Ethernet is also capable of supporting 1Gbps today over both fiber and the ubiquitous Category 5 (and up) twisted-pair copper wiring. But Ethernet has a better road map: The IEEE 802.3ae working group is hard at work on 10Gbps Ethernet and expects products to appear in 2002, and there is talk of 100Gbps Ethernet after that. (In an ironic twist, 10Gbps Ethernet could end up being based on Fibre Channel's new 2.5Gbps physical layer.) Overall, it's reasonable to expect Ethernet to eclipse Fibre Channel in terms of raw data rates by an order of magnitude in the next few years.

The "software" side is more accurately described as protocol performance and is one area where Fibre Channel will likely have an advantage for a while. Early iSCSI performance data indicates that initial implementations will achieve lower bandwidth utilization than Fibre Channel, but this can be expected to improve with time and experience. The performance of iSCSI over Ethernet may be less than that of SCSI over Fibre Channel for the near future simply because of the increased protocol overhead of encapsulating the data in IP datagrams and TCP segments to provide a "reliable transport." However, improvements in network protocol performance via methods such as hardware checksums and zero-copy TCP can be expected to narrow-or ultimately eliminate-the advantage that Fibre Channel holds in this area.

Again, history has shown that the simple and robust design of Ethernet can hold its own against faster competitors, even overtaking them after they have a significant lead. This happened when 10Mbps Ethernet beat 16MBps Token Ring and when 100Mbps Ethernet beat both FDDI and Asynchronous Transfer Mode (ATM), even though ATM was much faster until the recent advent of Gigabit Ethernet.


"Cost" is the category that will probably make the biggest difference over the long haul in determining what future SANs will look like. Because this is one of the high-weight items, even a small lead in this area becomes magnified as the marketplace determines the acceptance of a new technology. It's here that Fibre Channel, for all its strengths, will have the hardest time keeping up. Ethernet and iSCSI-based SANs have several things that make them less expensive-all ultimately related to the "dumb is cheap" maxim.

To start with, Ethernet has one thing going for it that makes it the 800 lb. gorilla from a cost point of view: volume. The popularity of Ethernet networks allows the silicon for these devices to be built in enormous quantities, making it very inexpensive. Entire embedded Ethernet Web-server devices are available now for less than $50, quantity one. That sort of price point is only achievable by leveraging the huge volume of Ethernet chips used elsewhere.

But there's more than just volume economics at work here. Ethernet's economic superiority is also due to its "dumb network" architecture, which pushes the intelligence out of the network components, allowing them to be simple and, consequently, cheap. By contrast, Fibre Channel is a more "elegant" and "powerful" architecture, but this comes at the price of vastly increased component complexity, which in turn drives up the cost of Fibre Channel components.

This same volume factor applies not only to host adapters, but also to other network components such as switches and routers. In fact, it's the cost differential of these devices that provides the most striking examples of Ethernet's cost superiority. I recently priced out a small no-frills 9-port Gigabit Ethernet switch for a test setup for less than $100 per port. A high-end managed Gigabit Ethernet switch could cost nearly 10 times as much, but that's still quite reasonable compared to Fibre Channel switches. Such devices aren't going to put McData out of business right away, but they do clearly point the way to an entirely different economic landscape for SANs-one that makes SAN ownership a reality for many that can't afford or justify SANs today. It is this attribute that will allow iSCSI/Ethernet to make its impact felt not only in the data center as a SAN replacement, but also to push all the way out to the end nodes-something that's generally not cost-effective with other technologies.

Finally, iSCSI/Ethernet will be much cheaper than Fibre Channel simply because it leverages the infrastructure and knowledge base already in place. For all the efforts at spreading the word and educating people, SAN design is still overly complex and arcane. IP and Ethernet network design, configuration, and maintenance are far better and more widely understood, and nearly every organization has existing expertise.

Again, history shows that it is exactly these attributes that allowed Ethernet to relegate Token Ring to a footnote in history: Not only were Ethernet adapters cheaper than their Token Ring counterparts, but the simplicity of Ethernet allowed network devices to offer much lower cost per port. This, in turn, allowed switching to penetrate LAN environments with a per-port cost that was orders of magnitude less than Token Ring switches. This all added up to a significant savings for companies that recognized that Ethernet was "good enough" to meet their LAN needs, especially since it enabled them to network things they could not have afforded to otherwise.


Although often discounted, ease of use and deployment is another of the most important attributes in determining technology winners. This is another area where Ethernet and IP shine over competitors. In fact, Ethernet has become the "Little Engine that Could" of the networking world. The fundamental simplicity of Ethernet has led it to become the network interface of choice for the home side of the DSL or cable interface, even as its engineering attributes are now positioning it for the other side of the same device as a "first-mile" interface. Few things in the IT world work as easily and predictably as plugging a computer into an Ethernet/IP network.

Compared to other networks, the simplicity and ease of Ethernet is remarkable. An example would be Ethernet versus ATM. Although ATM offers capabilities such as congestion control, QoS, and a design that is inherently better for handling isochronous traffic like voice or video, it is considerably more difficult to implement. This is one reason that the telco carriers and interconnects are the primary drivers of 10 Gigabit Ethernet, since setting up a SONET ring is relatively easy, but layering the ATM network on top of it is not.

Finally, Ethernet and IP have another advantage over Fibre Channel, in that they leverage the existing knowledge within an organization. IP SANs will stress the technology differently (in particular, network administrators will soon learn the true importance of latency as a linchpin of performance) but those that truly understand how to build a high-performance Ethernet LAN will be well on their way to doing the same in a SAN environment. Not only that, but the popularity of switching (and the fact that switching will be required for 10 Gigabit Ethernet) has embedded much of the "practice" of high-performance networking into the switches themselves, again making the design and deployment of such networks as easy as is possible today.

Looking ahead

Taken individually, each of these points can be less than persuasive. But they're not taken individually: Each is part of the decision process in determining the acceptance of a new technology, and they all build on one another.

The biggest strengths of the iSCSI/Ethernet alternative are in the areas that the market values the most. The advantages of Ethernet and IP in the storage networking environment are so compelling that they create a "technology dislocation" that will change the face of storage networking. This will be especially true if the economic benefits of the iSCSI/Ethernet duo become an alternative to FC-AL for disk and tape drive interfaces, which could conceivably work as either iSCSI targets or perhaps even individual network-attached storage (NAS) units, depending on the firmware. Although Fibre Channel won't vanish overnight, it will eventually stop growing and will be relegated to those networks already installed or the rarer variety that requires its added capabilities.

Although Fibre Channel is arguably a far better technology than either ATM or FDDI, we can expect it to suffer a similar fate in the future. Surely there will be other disruptive forces as well, but they will have an uphill battle against the mass and strength of a storage network based on well-known and well-proven technologies that also have the advantage of vast economies-of-scale.


InfiniBand is clearly another disruptive technology, but it may not impact storage networking as much as many people expect. First, it's important to understand that InfiniBand is a completely different "animal" than the networks discussed so far. InfiniBand will almost certainly succeed simply because of the mass of big systems vendors backing the standard. But it offers something very different: InfiniBand is a channel network, providing not only the connectivity required by devices and networks, but also the massively fast and wide memory channels required by clusters and other distributed computing approaches.

One of the big determinants of the ultimate success of InfiniBand in this space may be its suitability as a direct interface for storage devices. InfiniBand will probably have a huge impact, but unlike iSCSI, its impact will be mostly relegated to the confines of the glass house. There will be a battle in the data center, to be sure, but iSCSI's flexibility, economic advantages, and extensibility across network space tend to indicate that it is a winner that will be with us for a long time.

Dub Dublin lives in Austin, TX, and is vice president and general manager of Conservor (www.conservor.com), an on-site storage services provider headquartered in Sugar Land, TX.

This article was originally published on May 01, 2001