Going the distance with IP storage over WANs

Emerging standards such as FCIP, iFCP, and iSCSI may lower costs and make storage transfers over IP wide area networks affordable to more companies.

By Dave Simpson

As storage environments and storage area networks (SANs) grow, IT managers increasingly have the need to extend data transfers beyond the confines of the enterprise over longer distances such as metropolitan area networks (MANs) and wide area networks (WANs). This requirement is being fueled by an increased emphasis on applications such as disaster recovery, business continuance and, in general, data protection.

The ability to move SCSI commands and block-level data over long distances can benefit a number of related applications. For example:

  • Remote mirroring and data replication (asynchronous and synchronous), including automatic fail-over from a primary site to a "hot standby" sites or sites;
  • Resource consolidation (e.g., leveraging storage resources across multiple, geographically dispersed sites);
  • Migration of data between data centers or platforms, without impacting applications;
  • Remote tape-or disk-backup and restore;
  • Remote data access;
  • Multi-point content distribution;
  • Remote booting (e.g., diskless servers booting off primary site servers); and
  • Storage virtualization that extends beyond single-site SANs.

There is nothing new about transferring storage commands and data over long distances. IT organizations have for years been pushing storage traffic over WANs with products generally referred to as "channel extenders." This began in ESCON-based mainframe environments with products from vendors such as Computer Network Technology (CNT) and Inrange and more recently migrated to channel extension products for Fibre Channel SANs. Channel extenders often use technologies like SONET, ATM, and WDM/DWDM over expensive dedicated lines.

Those approaches are still good for high-bandwidth applications that require the utmost in deterministic, synchro nous data transfers where cost is not a primary consideration. But traditional channel extenders suffer from two serious drawbacks: They're extremely expensive, and they're proprietary, which leads to vendor lock-in and non-interoperability between products from different vendors. As such, those approaches typically have been limited to large enterprises with mainframe-size IT budgets.

Standards on the way

But a number of emerging technologies-and, more importantly, standards-promise to erase the drawbacks to traditional channel extension products and make long-distance storage transfers affordable to more companies. The three standards under development by the Internet Engineering Task Force (IETF) are FCIP, iFCP, and iSCSI. Although they differ in implementation and design goals, each will allow end users to extend their storage environments by running SCSI commands and block data over standard IP networks, potentially even the public Internet.

In an FCIP configuration, IP addresses and TCP connections are used only at the FCIP tunneling devices at each end point of the IP network.
Click here to enlarge image

Participants in the IETF standards committees hope to complete and ratify the specifications by mid-year. However, products based on "pre-standard" specifications are already available and will be compliant with the final standards shortly after ratification.

For example, Cisco is shipping the iSCSI-based 5420 Storage Router (resold by IBM) and Pirus Networks is shipping the iSCSI-based PSX1000 Storage Utility Switch. Nishan Systems has multi-protocol storage switches that support iFCP (developed primarily by Nishan), iSCSI, and FCIP. A variety of vendors are shipping boxes or cards that support FCIP, including Cisco, CNT (which also sells proprietary channel extenders), Lucent, SAN Valley, and SANcastle. And virtually all of those vendors, and many others, will eventually support iSCSI.

However, as the saying goes, "The good thing about standards is that there are so many to choose from." The existence of more than one standard for extending storage environments over long distances has led to confusion among end users.

FCIP extends Fibre Channel SANs

The Fibre Channel over IP (FCIP) protocol is designed to connect Fibre Channel SANs over standard TCP/IP networks, essentially creating one big SAN. (Native Fibre Channel-without extensions-has a distance limitation of 10km.) As such, analysts and vendors say that if you've invested in Fibre Channel (as have about 46% of companies with annual revenues of more than $1 billion), and you need to extend connectivity between those SANs, then FCIP-based products (bridges, switches, routers, or gateways) are a good solution. (For more information on SAN adoption rates relative to company size, see the cover story in this issue.)

"If you have a significant Fibre Channel infrastructure, FCIP is a good successor to older channel extension solutions," says Clod Barrera, technical strategy director for IBM's storage systems group. "But [as with all IP storage solutions] you still need to provide for sufficient bandwidth on the IP network."

FCIP devices typically have Fibre Channel ports on one side and Gigabit Ethernet ports on the other side. The devices encapsulate (rather than actually translate) Fibre Channel frames in TCP/IP frames and tunnel them over the IP network.

According to the book, IP SANs: A Guide to iSCSI, iFCP and FCIP Protocols for Storage Area Networks, by Tom Clark (Addison-Wesley), "FCIP is a means of encapsulating Fibre Channel frames within TCP/IP, specifically for linking Fibre Channel SANs over wide areas. Instead of a native IP connection between individual storage devices, FCIP provides IP connections between Fibre Channel SAN islands." An FCIP configuration is shown on p. 24.

iFCP supports Fibre Channel end devices by emulating an interface to an F_Port (Fibre Channel switch) connection.
Click here to enlarge image

FCIP can be implemented in a standalone box (gateway, switch, router, etc.), in a blade installed in an IP router, or via an FCIP port integrated into a Fibre Channel switch.

Compared to traditional channel extension products, FCIP devices are relatively inexpensive. For example, SAN Valley's 4-channel (Fibre Channel and Gigabit Ethernet) FCIP-compatible SL1000 IP-SAN Gateway is priced at $30,000.

Similarly, SANcastle's 2-port (Fibre Channel and Gigabit Ethernet) GFS-2 switch is priced at $15,000. The 8-port GFS-8 ranges from $26,500 to $37,000, depending on port configuration and features. These devices connect to Fibre Channel switches via the E_Port standard, essentially becoming part of the Fibre Channel SAN fabric.

Most of the major systems vendors do not actually manufacture FCIP (or iSCSI or iFCP) devices, preferring to partner with other vendors. For example, Compaq has qualified its SANworks Data Replication Manager software with IP storage products from vendors such as Cisco, CNT, Lucent, Nishan, SAN Valley, and SANcastle. IBM, for example, resells Cisco equipment and has partnerships with other vendors like CNT.

iFCP: IP for FC devices

According to Clark's book, "iFCP [Internet Fibre Channel Protocol] is a gateway-to-gateway protocol for providing Fibre Channel fabric services to Fibre Channel end devices over a TCP/IP network. Like FCIP and iSCSI, iFCP uses TCP for congestion control, error-detection, and recovery. Unlike FCIP, iFCP does not simply join Fibre Channel fabrics over distance, but can be used in place of Fibre Channel fabrics. Fibre Channel storage arrays, HBAs [host bus adapters], routers, switches, and hubs can be plugged directly into iFCP storage switches." The figure at the top shows a detailed layout of an iFCP configuration.

Although iFCP may have some technical advantages and can provide a transition to iSCSI, one potential drawback to iFCP is that it's championed primarily by only one vendor-Nishan Systems. (However, a variety of other vendors are participating in the IETF iFCP standards committee, which is chaired by a representative from Nortel.) As such, many analysts and vendors view iFCP as a dark horse in the standards race. To date, Nishan is the only vendor shipping iFCP-compliant products.

iSCSI: end-to-end IP SANs

Unlike FCIP, Internet SCSI (iSCSI) does not pre-suppose the existence of Fibre Channel SANs, although Fibre Channel devices and/or SANs can be used via gateway devices such as Cisco's 5420 router. Eventually, end users will be able to build end-to-end iSCSI SANs that include iSCSI target devices (e.g., disk arrays and tape libraries), iSCSI host adapters, and IP switches-all linked via standard IP networks (local or wide area). The figure on the left illustrates an iSCSI configuration.

"iSCSI is a new transport under the base SCSI protocol," explains Doug Ingraham, senior manager of product management in Cisco's storage technology group. "So it can be a native protocol for either hosts or target devices, or you can have a gateway, bridge, or router between iSCSI and, say, Fibre Channel."

Although IT organizations can use iSCSI gateway devices to link Fibre Channel or SCSI devices over IP networks, ideally an iSCSI configuration would entail an all-IP approach. "iSCSI is primarily a means for using an IP infrastructure to connect servers and storage resources," says Sandy Helton, executive vice president and CTO at SAN Valley, "and to do that, ideally all servers and target devices should have iSCSI interfaces, although you can use gateway devices."

Although gateway devices can be used to support existing Fibre Channel or SCSI devices, a native iSCSI storage network would consist of iSCSI initiators (e.g., host adapters) and iSCSI target devices (e.g., disk arrays or tape libraries).
Click here to enlarge image

As such, the use of iSCSI as a wide-area transport will partially depend on adoption of iSCSI at the local SAN level, where it represents a potential alternative to Fibre Channel. (For more information on iSCSI progress, see "iSCSI SANs inch closer to reality," InfoStor, January 2002, p. 1.)

"Adoption of iSCSI at the WAN level depends in part on adoption of iSCSI at the local SAN level," says Cisco's Ingraham. "So, at the WAN level, adoption of FCIP will precede adoption of iSCSI." However, it's likely that many IT organizations will deploy FCIP and iSCSI-and, possibly, iFCP-in conjunction with each other.

Because of the anticipated difference in adoption rates between FCIP and iSCSI, vendors like SAN Valley and SANcastle do not plan to incorporate iSCSI support until some time next year. Eventually, most players in this market are expected to support multiple standards and protocols in the same device.

Adoption of iSCSI at the local SAN level will depend in large part on technology developments on the host and target ends. So far, IBM is the only large vendor shipping an iSCSI-based disk array (the 200i), although others such as Compaq are expected to follow suit later this year.

On the host side, vendors such as Alacritech and Intel are already shipping iSCSI adapters with TCP/IP offload engines (TOEs), and other vendors (e.g., Adaptec, Agilent, Emulex, and QLogic) will follow within a couple quarters. On the software front, FalconStor Software's IPStor is compatible with iSCSI.

Another gating factor for IP storage acceptance-and iSCSI in particular-is that even though the core standards may be ratified this year, there is still a lot work to be done in areas like security and management.

In the case of both management and security, the goal of iSCSI proponents is to leverage existing TCP/IP mechanisms as much as possible. "We want iSCSI to add as little as possible to users' existing IP protocols, including security, authentication, and discovery protocols," says Cisco's Ingraham.

"Basic management functionality is included in the iSCSI spec, such as iSNS discovery mechanisms," says IBM's Barrera, "but management is a very big topic and things like Quality of Service will have to be addressed." (For more information on iSNS, see "iSNS: Technical overview of discovery in IP SANs," Parts 1 and 2, InfoStor, November 2001 and January 2002.)

Clark sums up the positioning of the three IP storage protocols: "FCIP represents Fibre Channel perpetuation, while iFCP represents transition from Fibre Channel to IP storage networks, and iSCSI represents the ultimate goal of homogeneous IP storage networking."


As with any long-distance data transfers-whether networking or storage-there's a price to pay: latency. Because you're limited by the speed of light and other factors, the longer the distance, the higher the latency. For many network applications, this overhead doesn't pose a problem, but for many storage applications, latency can pose a huge problem.

For example, heavy transaction-oriented applications with a lot of write operations such as databases are typically not good candidates for IP storage over WANs because they can't afford to suffer the latency delays. On the other hand, applications such as backup and asynchronous replication or mirroring may not suffer significantly from latency delays.

In addition to distance and the speed of light, latency depends on a number of other factors, including the network infrastructure (e.g., type and speed of the transport), number of hops, buffer credits, performance of the gateway devices, and TCP/IP itself.

In the case of lost packets, TCP/IP re-transmits those packets. So the longer the distance and the more latency on the WAN, the more that packet loss will affect overall performance.

This is related to the basics of SCSI operations. For any SCSI transaction, there is a request (command), followed by a data transfer and a response. Subsequent transactions to the same drives or LUNs cannot be completed until the prior ones are completed.

Recommendations for minimizing latency include

  • Minimize the amount of packet loss (e.g., "shape" Fibre Channel traffic to the speed of the WAN link to minimize dropped frames and congestion);
  • Maximize the use of available bandwidth;
  • Use devices with a lot of buffering; and
  • Use parallel (as opposed to serial) data streams between the source and destination. This can be accomplished with software such as EMC's SRDF and IBM's PPRC.

(For more information on latency, in particular, advice on how to minimize latency, see "Bandwidth vs. latency in SAN extensions, InfoStor, December 2001, p. 26.)

Expect slow adoption

Despite all the hype and activity around iSCSI and IP SANs, acceptance is expected to be slow. For example, International Data Corp. expects Fibre Channel to account for more than 99% of all SAN switch port shipments this year, with Ethernet/IP accounting for less than 1%. However, by 2005, IDC expects Ethernet/IP to account for about 17% of all SAN switch port shipments, with Fibre Channel holding on to about 76% and InfiniBand garnering about a 7% market share.

Even strong iSCSI supporters such as IBM predict that iSCSI's impact on Fibre Channel will be minimal for the foreseeable future. "iSCSI can be used as a replacement for Fibre Channel in SANs," says IBM's Barrera, "but we don't think end users will replace Fibre Channel with iSCSI for quite some time because Fibre Channel is still ahead in areas such as performance [2Gbps vs. 1Gbps] and management software."



For more information about IP storage, see the following articles from InfoStor:

"iSCSI SANs inch closer to reality," January 2002, p. 1.
"Fibre Channel and IP SANs: An independent view," January 2002, p. 48.
"IP storage and Fibre Channel: Competition and co-existence," December 2001, p. 21.
"Bandwidth vs. latency in SAN extensions," December 2001, p. 26.
"Using DWDM for storage networking in MANs," December 2001, p. 32.
"IP storage: Separating hype from reality," December 2001, p. 48.
"iSCSI: Storage networking for the Internet generation," November 2001, p. 42.
"What is the role of IP storage in SANs?", November 2001, p. 64.
"Demystifying iSCSI, iFCP, mFCP, FCIP, and iSNS," September 2001, p. 66.

IP SANs: The book

Click here to enlarge image

The most comprehensive book on IP SANs (both local and wide area) is Tom Clark's recently released IP SANs: A Guide to iSCSI, iFCP and FCIP Protocols for Storage Area Networks (Addison-Wesley, $42.99, ISBN: 0-201-75277-8). Although the primary emphasis is on the three IP storage standards that are in development in the Internet Engineering Task Force (IETF), the book also provides a wealth of information on topics such as shared storage, storage networking (including SAN and NAS), security, and Quality of Service (QoS). The text also provides in-depth reviews of the SCSI, TCP/IP, UDP, and InfiniBand protocols.

The book is written for both network professionals (who may not have a thorough understanding of SCSI) and storage professionals (who may not have an adequate understanding of TCP/IP). IT managers may be particularly interested in the section on IP storage applications, which cuts through the various technologies and focuses on real-life problems/solutions.

Clark's text also tackles the primary controversy in this space-Fibre Channel vs. IP-in a somewhat even-handed manner. However, as noted in the book's introduction, his prejudices do show up in the discussion of the three protocols: iSCSI, iFCP, and FCIP. It should be noted that Clark is director of technical marketing at Nishan Systems, the primary proponent of iFCP.

Alternatives for storage over WANs

Although there is a clear trend toward relatively low-cost, IP-based approaches to extending storage environments based on standards such as FCIP and iSCSI, a number of other alternatives exist, and they may provide advantages in high- performance applications over very long distances.

Computer Network Technology (CNT) is one of the few vendors pursuing virtually all of the options. The company has provided high-end, proprietary "channel-extension" routers for years, has recently shipped an FCIP-based edge router, and in the future, will support iSCSI.

Brian Larsen, senior director of product management at CNT, argues that FCIP and iSCSI boxes may be less expensive, but that proprietary approaches provide higher performance and better data flow management via techniques such as store-and-forward technology, sophisticated buffering and caching, and compression. For example, CNT claims compression ratios ranging from 2:1 to 15:1, which effectively increases available bandwidth and decreases network costs.

"New boxes may be less expensive, but those costs can pale in comparison to the network costs," says Larsen.

There are many other alternatives for extending storage applications over MANs and WANs. For example, Akara Corp., which competes with CNT, advocates the use of "Storage through SONET" for "more-stringent" applications and distances than can be adequately handled by products based on standards such as FCIP and iSCSI (see table). Although Akara's storage extension multiplexers can replace FCIP or iSCSI devices, they can also be used in conjunction with those boxes, encapsulating Gigabit Ethernet traffic and running it over SONET, DWDM, or dark fiber connections. In addition to Fibre Channel and Gigabit Ethernet, Akara supports other storage protocols such as FICON and ESCON in its Optical Utility Services Platform (OUSP).

Storage extension technologies
 Storage though SONETFCIPiFCPiSCSI
Capacity5 to 1000MBps5 to 100MBps5 to 100MBps5 to 100MBps
Distance extensionLimited by latencyLimited by latencyLimited by latencyLimited by latency
Primary applicationsAnyRemote SAN connectionHost-targetHost-target
SecurityPhysical isolationVia IPSecVia IPSecVia IPSec
DrawbackOptical onlyRequires external switch;
indeterminate latency
Requires external switch;
indeterminate latency
Requires external switch;
indeterminate latency
Source: Akara

This article was originally published on March 01, 2002