A realistic view of IP-based SANs

A number of storage-over-IP initiatives exist, but tall technology hurdles remain.

By Tom Clark

The vast majority of storage area network (SAN) products installed over the past four years are based on Fibre Channel technology. Fibre Channel has become synonymous with SANs because it is the first architecture to efficiently resolve the bandwidth, scalability, distance, and protocol requirements of storage applications.

Traditional parallel cabling schemes have kept pace with growing bandwidth demands, but have lacked the scalability, distance, and flexibility required to share storage resources in enterprise networks. Gigabit Ethernet overcomes those limitations, but has lacked the upper protocol support for reliable transport of large blocks of data to disk. The processing overhead of TCP/IP over Ethernet has been a key inhibitor for the adoption of traditional networking in SAN implementations.

Even at 10Gbps, a large pipe alone cannot compensate for the inefficiencies of a protocol that swamps server resources. In addition, given the value of an enterprise's information, the reliable transport of data to disk presents a significant challenge to any networking architecture. As a protocol developed for small messaging traffic over an unstable network, TCP/IP has not been the first choice for mission-critical storage applications.

Lucent and Vixel are collaborating on FC/IP to facilitate high-speed WAN links to transport Fibre Channel SAN traffic. The technology wraps Fibre Channel frames in IP and uses IP routing to carry the SCSI/FC payload from source to destination across the WAN.
Click here to enlarge image

Several new initiatives, however, are promoting the development of storage networking on IP and Ethernet. For these solutions to be viable, several obstacles must be overcome, including bandwidth, protocol overhead, and reliable transport of large blocks of data required for storage. These are not trivial problems to resolve, and despite the marketing wrapped around some of these efforts, stable and interoperable products are still some time away.

When these products are available, they will provide useful new options for SAN design, particularly for wide-area applications. The combination of Fibre Channel- and IP-based SAN products will enable administrators to link storage applications throughout the enterprise, from data centers to regional and remote sites, in a common infrastructure.

SCSI over X

The SCSI protocol is defined in the ANSI SCSI Architecture Model (SAM-2). The model de taches SCSI as a transport protocol from any par ticular network topology, and, so, is the base document for porting SCSI to Fibre Channel, IP/Ethernet, or other transports. The SCSI protocol provides commands and status exchange methods for establishing sessions between servers and disks (or other targets) and for efficiently moving large blocks of data between them.

Because SCSI is still the most optimized protocol for block data transport and requires minimal CPU overhead, it has survived as the universal language of storage applications. Given its tight integration in operating systems, there is little incentive to replace SCSI with something else. Instead, the focus has been on leveraging SCSI's proven abilities and accelerating SCSI communications on higher-speed networks, including Fibre Channel, Gigabit Ethernet, and eventually InfiniBand.

For Fibre Channel, serial SCSI (SCSI-3) is mapped at the upper layer protocol (ULP) level. In the Fibre Channel model, FC-4 is responsible for translating between the operating system's SCSI addressing for disks and other storage targets and the underlying Fibre Channel transport protocols. SCSI requests (e.g. writing a file to disk) are handed down by the operating system to the Fibre Channel drivers in a host bus adapter (HBA), which in turn handles the association between SCSI bus/target/LUN addressing and Fibre Channel network addressing. The more subtle issues of frame assembly for transport, sequencing, media access, error detection and recovery, etc., are resolved by Fibre Channel's lower layers.

Substantial intellectual effort has been invested in solving the complexities of reliably moving SCSI data at Gigabit speeds. Witness the mountain of documents generated by the ANSI T11 workgroups, the Fibre Channel standards themselves, and the ongoing activity of the Fibre Channel Industry Association (FCIA) and the Storage Networking Industry Association (SNIA).

While the Fibre Channel vendors have solved the most critical issues for safely transporting SCSI data at high speeds, Fibre Channel offers no direct means for linking remote sites that are more than 10km apart. Because applications such as disaster recovery may require regional or national links of much longer distances, local sites using Fibre Channel SANs need intervening communications devices.

In a few years, some environments will combine Fibre Channel SANs, Ethernet backbones, and IP networks via routers and bridges.
Click here to enlarge image

Dense wave division multiplexing (DWDM) is one solution because it enables native Fibre Channel signaling to span wide areas. A complementary ap-proach is to wrap Fibre Channel frames in IP and use IP routing to carry the entire SCSI/FC payload from source to destination across the WAN. Lucent and Vixel, for example, are collaborating on FC/IP to facilitate high-speed WAN links to transport Fibre Channel SAN traffic. The advantage of this method is that minimal protocol conversion is required, while the data integrity features (e.g., cyclic redundancy checking, or CRC) and flow control of Fibre Channel are maintained. FC/IP must still address the inherent issues of IP routing, but because IP is only required for the wide-area commute, end devices at the edge sites enjoy the reliability of Fibre Channel without modification. Products based on FC/IP for wide-area applications are appearing in the market much sooner than those based on the more problematic SCSI/IP architecture.

Because the SAM-2 model does not dictate the physical transport, with sufficient investment and clever engineering, SCSI can theoretically be mapped to any conveyance. The success of Fibre Channel-based SANs to date, and the projected billions of dollars in market opportunity for storage networking over the next few years, has stirred the interest of other players, including those heavily invested in IP routing.

Cisco, for example, began promoting the concept of "IP SAN" in late 1999 and is supporting concurrent SCSI-over-TCP/IP initiatives before the Internet Engineering Task Force (IETF).

Advocates of carrying the SCSI protocol over TCP/IP promote the One Big World theory, which maintains that TCP/IP has become the Esperanto of data communications and all citizens must be joined together by TCP/IP routers. Having shown their enormous potential market value, SANs are natural targets for IP.

SCSI/IP initiatives include:

  • Adaptec's SCSI Encapsulation Protocol (SEP/IP), which uses a less labor- intensive approach than TCP.
  • SAN Ltd.'s use of the Service Specific Connection Oriented Protocol (SSCOP) over IP for more reliable link-level transmission.
  • Genroco's use of Scheduled Transfer Protocol over IP (STP/IP), which also offers reliable end-to-end transmission.
  • Nishan's Storage Over IP (SoIP), a term that is trademarked. Although Nishan is submitting SoIP to the IETF for standardization, it is difficult to understand how an Registered will fit into an open-systems environment.

IBM/Cisco: iSCSI

Currently, the main SCSI/IP project in the IETF is the IBM/Cisco iSCSI (Internet SCSI) draft. The work behind this draft has focused on adapting the requirements of SAM-2 to TCP/IP. Because the participants in the draft bring expertise in both the SCSI protocol and IP networking, the exchange conducted on the iSCSI reflector reveals an acute awareness of the challenges ahead.

The hurdles include, but are not lim ited to, mapping SCSI bus/target/LUN addresses to something more intuitive than a dotted decimal IP address (in this case, a URL-friendly format), SCSI session establishment and termination via TCP, dealing with exceptions at the command level, and overcoming the relative weakness of TCP/IP's checksum in lieu of Fibre Channel's more rigorous CRC checks.

For the sake of expediency, some issues have been set aside (e.g., the problem of streaming video over iSCSI, which wants nothing to do with TCP's retransmission algorithms). Other storage-specific areas, such as SCSI Extended Copy (or third-party copy), pose difficulties due to addressing and authentication requirements. Still, with sufficient time and engineering resources, the iSCSI adaptation for TCP/IP should eventually result in one of several viable solutions for IP transport of storage data, at least at the SCSI protocol level. However, this still leaves several significant issues to resolve.

SCSI/IP assumptions

In addition to mapping the SCSI protocol to TCP/IP or other IP variants, successful implementation of SCSI/IP for storage networking will require resolving bandwidth, overhead, data integrity, and security issues. Putting SCSI over IP will have little value if it remains signifi cantly slower than other technologies, consumes server cycles, and cannot ensure reliable delivery.

To address bandwidth concerns, proponents of SCSI/IP link it to the widespread availability of 10Gigabit Ethernet (10GbE). The only problem with this assumption, however, is that although 10GbE has been demonstrated at trade shows, it is not yet a widely available production-level transport. As with any new technology, the initial per-port cost will restrict deployment to the upper end of the market, where early adopters also inevitably assume some risk.

IT organizations do not typically embrace this type of environment for their most mission-critical data. Fibre Channel, for example, did not see a significant ramp in adoption until basic stability and interoperability issues were resolved, and until per-port costs fell in line with perceived value.

If SCSI/IP requires 10GbE to show significant advantages in throughput, the physical transport, not the protocol development, will be the gating factor for IP-based SANs.

In addition to 10GbE, reducing the protocol processing overhead of TCP/IP is a prerequisite for successful SCSI/IP deployment. With TCP/IP's consumption of more than half a server's CPU cycles at gigabit speeds, no one will implement an entire storage network with such an enormous appetite. The resolution is to offload TCP/IP processing into dedicated silicon via ASICs. This is a significant development for the IP world, because it potentially accelerates processing for a wide variety of applications. As with 10GbE, cost and availability will drive the rate of adoption, as will demonstrated interoperability among multiple vendors.

Resolving data integrity and security issues will also be essential for IP-based storage networks. Data integrity requires reliable mechanisms to validate data conformity at the packet level via CRC checks, as well as rigorous flow control to ensure that packets are not dropped. Occasional corruption of e-mail may be acceptable in IP environments, but not corruption of user information written to disk.

Data security is a particular problem when the economies of SCSI/IP are link-ed to transport over the Internet. Real- istically, enterprises will probably elect to use private links for storage traffic, while the temptation to use the Internet (e.g., for smaller, remote sites) must be balanced with the usual security concerns the Internet presents.

Storage peripherals and SCSI/IP

Although the most obvious use for IP-based storage transport is for wide-area applications, some SCSI/IP vendors are promoting end-to-end IP-based SANs. This has profound implications for disk and tape manufacturers who, in addition to traditional SCSI interfaces and Fibre Channel, would also have to support native 10GbE interfaces (and, potentially, InfiniBand) and SCSI/IP processing logic. With the future deployment of 10GbE and reduction of TCP/IP overhead via silicon, the components to accommodate IP-based SANs would be available, but at the expense of addi tional product development, support, and extensive interoperability testing. Having just completed this exercise with Fibre Channel over the past four years, it is unlikely that the disk and tape vendors are anxious to be first in line for SCSI/IP interfaces.

A transitional strategy would preserve Fibre Channel at the storage peripheral, while providing Fibre Channel-to-10GbE conversion at the Fibre Channel switch or IP router. This places the burden of bridging tech nologies onto the interconnect vendors that have signed up to provide SAN connectivity.

Marketing vs. reality

Unfortunately, the emergence of SCSI/IP initiatives has been accompanied by volleys of FUD launched by both the SCSI/IP and Fibre Channel camps. The smell of potentially billions of dollars in revenues is feeding extremist behavior that may unintentionally undermine customer confidence in storage networking as a whole.

Marketing material offered by some SCSI/IP proponents declares the death of Fibre Channel SANs, citing interoperability and management issues. At the same time, some Fibre Channel proponents overzealously denounce the inefficiencies of IP and Ethernet based on outdated information.

Other SCSI/IP advocates speak as if viable, standards-based SCSI/IP products are just around the corner, although the standardization process alone guarantees that truly standards-based, interoperable products will take several years to emerge. One SCSI/IP vendor has already declared its proprietary SCSI/IP implementation to be a de facto standard, which ironically can only retard SCSI/IP development. In the end, this infrastructure infighting is a dispute among the plumbers over which pipes are better, when all the customers want is running water.

The reality is that both Fibre Channel and SCSI/IP over 10GbE (as well as fu-ture InfiniBand solutions) will provide useful tools for building enterprise storage networks. Products that bridge these technologies will provide the flexibility for installing the appropriate transports based on actual application requirements.

The reality is that Fibre Channel SAN solutions exist today, and few IT organizations can choose to delay addressing their storage problems for an additional one to two years while SCSI/IP and a stable 10GbE are under development. With an ever-increasing installed base of Fibre Channel SANs, it is in the interest of SCSI/IP vendors to incorporate Fibre Channel interfaces into their product roadmaps. And with the obvious benefits of IP-based storage solutions for WANs, it is in the interest of Fibre Channel vendors to embrace SCSI/IP and FC/IP solutions to satisfy end-user requirements.

There is a real danger that an unconstrained marketing dispute between competing technologies will create customer confusion and retard the adoption of storage networking. Customers generally do not buy a technology, but a business solution. That solution must demonstrate significant advantages over previous methods and, in today's open-systems environment, it must be standards-based and interoperable.

Umbrella organization

The proper forum for accelerating standardization and interoperability of all storage networking technologies is an umbrella organization, like SNIA, that promotes the common interests of all vendors. Because SNIA working groups encourage participation by all members, including vendors and end-users, developers of particular technologies can benefit from the input of their peers and be more closely aligned to customer needs. The SCSI Extended Copy standard that the iSCSI draft is attempting to map, for example, is a product of the SNIA backup working group.

The ability of SCSI/IP implementations to provide storage solutions for wide-area and other applications will depend on resolving bandwidth, processing overhead, and data integrity issues. The alignment of 10GbE, TCP/IP in silicon, and more rigorous data safeguarding, along with the standardization of SCSI mapping to TCP/IP, is only the first step toward a viable solution.

Products must undergo extensive interoperability testing to demonstrate standards compliance, must be affordable and available, and must establish a proven track record of supporting upper-layer storage applications such as backup and clustering. All of these things take time.

In the meantime, IT organizations will continue to address today's storage needs with today's optimum solution: Fibre Channel SANs. As both SCSI/IP products and Fibre Channel-to-Ethernet bridging products become available, IT organizations can extend their SANs with an even richer tool set to solve local and wide-area storage requirements.

Tom Clark is technical marketing director for Vixel Corp. He is also a board member of the Storage Networking Industry Association, co-chair of the SNIA Interoperability Committee, and the author of Designing Storage Area Networks: A Practical Reference for Implementing Fibre Channel SANs (Addison Wesley Longman).

This article was originally published on October 01, 2000