Drawing the lines on IP storage networking

End users need to know what their options will be as well as the potential advantages and disadvantages of the various proposals.

Announcements of new storage connectivity options this year have provided more promise and controversy than the industry has seen for years. One of the most intriguing promises is storage networking on standard IP networks.

While pure Fibre Channel vendors would probably prefer to limit the use of IP storage networking to wide area network (WAN) links, IP networks also include Gigabit Ethernet and, soon, 10Gbps Ethernet LANs, which may be adaptable for native IP storage area networks (SANs).

The emotions surrounding native IP SANs run high. Fibre Channel vendors view it as threatening, while others (such as Cisco, IBM, Adaptec, Hewlett-Packard, EMC, and Nishan Systems) have embraced native IP SANs as a way to establish leadership in a new high-growth market.

The use of Gigabit Ethernet and IP with Fibre Channel SANs appears to be inevitable, and poses a dilemma for Fibre Channel vendors. Clearly, there is a user requirement for supporting applications such as remote backup over a standard IP WAN, but the incumbent Fibre Channel SAN vendors want to avoid competing with the larger and more powerful Ethernet/IP vendors.

The dynamics of the situation have created a confusing environment for IT professionals who are faced with conflicting claims from the vendor camps. This article attempts to clarify the situation with an analysis of the positions (and motivations) of the vendors involved, and the potential strengths and weaknesses of the various approaches under consideration.

In general, there are two main camps in IP storage networking: tunneling and native IP storage. For the most part, the Fibre Channel SAN vendors are entrenched in the storage tunneling camp, while vendors such as Adaptec, Cisco, IBM, and Nishan are leading the charge in the native IP storage camp. Other vendors appear to be hedging their bets by playing in both camps.

Storage tunneling

Storage tunneling uses encapsulation to place frames from a Fibre Channel SAN inside the packets of an IP network for transport to another, distant Fibre Channel SAN. Upon arriving at their destination, the Fibre Channel frames are stripped of their encapsulation and placed on the distant SAN (see Figure 1). In this approach, Fibre Channel operates as an overlay network, not as an integral part of the IP network.

Because it simply provides a dedicated, point-to-point link between two SAN "islands," storage tunneling is functionally equivalent to dedicated dark fiber links-except it has more latency and, for the implementations announced so far, severe performance limitations. But dedicated dark fiber links typically are much more expensive and are not always readily available. By using IP, storage tunneling enables the use of existing metro area networks (MANs) and WANs. In other words, storage traffic can go anywhere an existing network goes. IP MANs and WANs cover a much broader range of geographical options and are less costly.

Figure 1: Storage tunneling uses encapsulation to place frames from a Fibre Channel SAN inside the packets of an IP network for transport to another, distant Fibre Channel SAN.
Click here to enlarge image

Several announcements have been made this year about storage tunneling products. In April, Computer Network Technology (CNT) announced tunneling for interconnecting its products over IP networks.

In May, Gadzoox Networks announced a storage tunneling product that is based on technology acquired with SmartSAN. Gadzoox has not yet announced a shipping date for its product. Although there still is no standard for storage tunneling, Gadzoox, with assistance from Lucent Technologies, is attempting to create one (see InfoStor, June 2000, p. 1).

From the standpoint of market clout, the most significant announcement for IP tunneling occurred in June when Brocade Communications and Cisco Systems an-nounced a technology partnership to work on a solution for mid-2001 delivery (see InfoStor, June 2000, p. 1). This agreement calls for Cisco and Brocade to jointly develop a Fibre Channel port on Cisco's Catalyst 6000 switches. The port will interoperate with an E-port on Brocade Silkworm switches and, presumably, with E-ports on other vendors' Fibre Channel switches.

Cisco will provide the protocol encapsulation and IP routing technology for transferring storage data over an OC-3 (155Mbps) WAN port. Although it does not move Cisco into the SAN market, the partnership does give Cisco an indirect presence by associating with Brocade, the current SAN switch leader. In addition, it may help Brocade stall the progress of the tunneling standard proposal sponsored by its rival, Gadzoox.

While at first glance the Cisco-Brocade partnership seems reasonable, it actually may contain the seeds of its own destruction. Brocade protects its data center SAN position by limiting Cisco's role to a MAN and WAN solution provider. Conversely, Cisco's commitment to the relationship is suspect, because the company has clearly communicated its intent to develop native IP storage. Cisco's current native IP storage proposal does not provide Fibre Channel compatibility. That would limit Brocade's participation in Cisco's future SAN business.

Beyond the business issues, there are also technical problems with storage tunneling. One such problem is the management of the tunneled traffic. With storage tunneling, the Fibre Channel devices communicate not as an integral part of the IP network, but as an overlay network. Consequently, the IP network and the Fibre Channel SAN islands require separate network management systems.

Familiar IP management tools, such as directory services, traffic control, and diagnostics, can be used only on the tunnel, rather than end-to-end at the device level. That, in turn, limits network administrators' ability to effectively manage the network. For example, asynchronous tape backup and semi-synchronous disk mirroring would have to be treated equally, resulting in poor overall network performance.

The storage tunneling products announced so far support a small fraction of the bandwidth that SANs can use.

Figure 2: Native IP storage provides a homogeneous network, built around a core of Gigabit and 10Gbps SANs.
Click here to enlarge image

That itself is a serious challenge to many storage network applications. For example, a file transfer operation that takes two to three hours over a Fibre Channel SAN can take more than 14 hours between SANs at the OC-3 data rate. When these discrepancies are applied to applications such as remote backup or vaulting, the intended goal is still out of reach. The tunnel simply may not be able to support the amount of data that needs to be moved during the prescribed backup window.

Finally, if you believe that native IP storage is the best approach in the long run, storage tunneling is not going to get you there. Storage tunneling does use IP networks, but it locks customers into Fibre Channel technology for their devices, subsystems, and SAN switches. That's great if you have a vested interest in Fibre Channel, but if you are an end user trying to move to native IP storage, it is a costly side trip.

Native IP storage

Historically, IT managers have chosen standard IP infrastructure products and services across their organizations. There is no reason to think that, given the option, they won't do the same for their storage networks. This does not mean they will combine storage network traffic and LAN traffic on the same physical network. Instead, it means they will use the familiar IP technology to develop discrete native IP SANs. The skills they have developed to control and manage IP networks will be applied to their new native IP SANs. Because IP and Ethernet have become de facto standards for all other forms of data transmission, there are more management tools and trained staff available to install and operate networks that are based on these technologies.

Gigabit Ethernet switches and routers have capabilities that previously were available only with Fibre Channel switches. These include ultra-low error rates, flow control, link aggregation, and full-duplex operation. Given these developments and IT organizations' preference for IP infrastructure products, Gigabit Ethernet is a serious contender for carrying native IP storage traffic.

Three approaches are being developed for implementing native IP storage, each implementing some version of serial SCSI, such as the upper layer of Fibre Channel's Fibre Channel Protocol (FCP) over an existing layer in the IP protocol stack. These approaches are discussed briefly below.

Adaptec's EtherStorage

Adaptec announced its Ether-Storage technology earlier this year (see InfoStor, June 2000, p. 1). EtherStorage introduces a new protocol, called Storage Transfer Protocol (STP), on top of IP and at a peer level with TCP and UDP.

By avoiding TCP, EtherStorage avoids TCP's latencies and heavy processor loading. While this could be very good for local SANs, EtherStorage will be incompatible with switches and routers that manage flows and priorities at the TCP level.

Adaptec plans to deliver EtherStorage products by mid-2001. In keeping with the company's heritage, EtherStorage will offer compatibility with parallel SCSI, but not with Fibre Channel.

IBM + Cisco = iSCSI

Another initiative receiving a fair amount of attention this year is an IETF draft proposal by IBM and Cisco to create a storage networking technology that uses TCP/IP. Referred to as iSCSI, the proposal still is at the draft stage, but is being worked on by representatives from IBM, Cisco, EMC, Hewlett-Packard, Adaptec, and others.

iSCSI uses a serial SCSI protocol that is said to be analogous to FCP. In other words, it will be like FCP, but not compatible with Fibre Channel or parallel SCSI devices. Because it uses TCP/IP, iSCSI will most likely work with all networking equipment.

The group working on iSCSI no doubt has the best intentions to finish the work quickly. However, the competitive nature of the group and the past history of standards processes would indicate that this specification could be a long time in the making.

With no expected release date for products based on an unfinished specification, end users looking for native IP storage based on iSCSI could wait a long time.

Nishan's SoIP

In late May, startup Nishan Systems announced plans to introduce a new technology called Storage over IP, or SoIP (see InfoStor, June 2000, p. 16). A third approach to native IP storage, SoIP uses a storage adaptation layer for the serial storage protocol over the IP stack. Nishan claims that SoIP can use either of the standard Layer 4 transmission protocols, making it compatible with all networking equipment.

Because it also is based on the FC-4 layer of the FCP standard protocol adopted by the Fibre Channel industry, SoIP is compatible with both legacy Fibre Channel and parallel SCSI devices. Nishan and its partners expect to ship SoIP-compliant products by year-end.

Click here to enlarge image

According to Nishan, one of the most significant aspects of SoIP is that it accommodates both Fibre Channel and SCSI interfaces, as well as native SoIP interfaces. In other words, an SoIP-based SAN will be able to incorporate existing Fibre Channel and SCSI storage devices while facilitating the construction of new native IP SANs. Future servers and storage subsystems could have Gigabit or 10Gbps Ethernet SoIP network interfaces, even though the existing server and storage interfaces continue operating with Fibre Channel or SCSI. Most importantly, data residing on existing subsystems does not have to be converted to accommodate the implementation of SoIP, and all existing SAN management applications will continue to run without modification.

Each of the three native IP storage approaches has the advantage of end-to-end network management. This will be a key benefit for IT organizations with multiple sites linked by MANs or WANs. Network administrators will be able to prioritize traffic to the device level, including the ability to reserve bandwidth using familiar flow management facilities available with many switches and routers. Native IP storage networks will be capable of transferring data at optimal data rates over LANs, MANs, and WANs. No longer SAN islands tied together by a foreign network, native IP storage provides a homogeneous network, built around a core of Gigabit and 10Gbps SANs (see Figure 2).

This homogeneous native IP SAN will be especially appealing to the new storage service providers (SSPs) that are building storage hosting centers.

By letting subscribers access their data over standard high-speed IP networks, SSPs could extend their own native IP SAN to the subscribers' data centers without having to install dedicated dark fiber.


Transporting data between SAN islands, storage tunneling uses IP networks for SAN-to-SAN connectivity and has cost advantages over dedicated dark fiber. However, the burden of supporting two separate management systems is not likely to meet with glowing approval from the IT community, which typically demands levels of integration that are greater than storage tunneling can provide.

While storage tunneling is attractive to Fibre Channel SAN vendors, it's not necessarily interesting to the IP industry. IP vendors will prefer to get a much larger slice of the pie, by participating not only in the metro and wide area storage networking markets, but in the lucrative local storage networking market as well.

The Brocade-Cisco partnership would appear to be a win for both companies, although Cisco is unlikely to put much muscle into this endeavor while also developing native IP storage. Regardless of Cisco's and Brocade's motivations, their combined market presence could derail the plans of storage tunneling competitors.

Meanwhile, native IP storage recently has generated more interest within the storage industry than anything else has generated in many years. It appears that users of storage networking are primed to give it a go, once there is something reasonable to try.

Adaptec's EtherStorage is promised for mid-2001. Having no backward compatibility with Fibre Channel, EtherStorage will start from scratch as a SAN infrastructure technology. Adaptec's selection of a proprietary Layer 4 protocol may be a good engineering decision, but it is likely to hinder market acceptance of the resulting products.

iSCSI, while supported by standards activities and a number of influential companies, is not likely to result in any shipping products until well into 2001, or later. Although it still is evolving, in its current form it will have no backward compatibility with Fibre Channel or SCSI devices.

SoIP is the only alternative slated for availability this year. If Nishan and its partners can deliver SoIP products that work reasonably well, end users are likely to start testing this year, and deploy SoIP SANs in limited configurations next year, just as they have been doing for the past few years with Fibre Channel.

Marc Farley is vice president of marketing at Solution-Soft Systems Inc. (www.solution-soft.com) in San Jose. He is also the author of (Osborne/McGraw-Hill, 2000).

Cisco-NuSpeed implications

In late July, Cisco Systems announced its intended acquisition of NuSpeed Internet Systems (see story on p. 1). If completed, this acquisition will have a significant impact on the storage networking market by signaling Cisco's intentions to fully participate.

However, NuSpeed's technology may not be ready for market for some time. Compared to billion-dollar valuations Cisco gave to other companies last year, the $450 million Cisco allocated to NuSpeed is relatively small. This probably indicates that the technology needs more time before Cisco is comfortable releasing it.

Beyond the posturing that Cisco, Brocade, and others may make, this move by Cisco seems to be a direct challenge to the entire Fibre Channel industry. The NuSpeed acquisition also raises questions regarding Cisco's partnership with Brocade. It will be interesting to see whether or not the acquisition will become part of that relationship through cross-licensing or other development agreements. It seems more likely that Cisco wanted to make a move to be independent of Brocade in the area of storage over IP.

Cisco's acquisition of NuSpeed may also have an interesting effect on Adaptec and Nishan Systems. Perhaps Cisco felt the need to defend its IP turf against these companies' IP storage developments. If so, the opposite effect could occur, with both EtherSCSI and SoIP benefiting from the attention drawn to this technology by Cisco.

This article was originally published on September 01, 2000