What is the role of IP storage in SANs?

Separating hype from reality, with an eye on geographically distributed network storage.


Walt Hinton
ManagedStorage International
Click here to enlarge image

Storage area networks (SANs) have come a long way from the days of proprietary mainframe channel extension. ESCON, IBM's proprietary protocol for mainframe-to-peripheral device communication, was arguably the first SAN, delivering many of the benefits associated with Fibre Channel SANs today. ESCON increased both I/O speeds and distances between hosts and associated devices. Its major benefit was the ability to reduce costs and improve uptime. Cost reduction was enabled by a switch-based architecture that allowed devices to be shared between hosts. Uptime was increased through the ability to switch multiple paths between hosts and devices.

In recent years, Fibre Channel SANs have taken the spotlight, promising greater performance and connectivity distances. But, the real benefits of SANs are the same fundamentals of improved uptime and lower costs through shared devices. Many of today's SANs were built to help re-centralize sprawling distributed client/server computing environments.

While the SAN market has not developed at the pace expected by some industry experts, there is no question that SANs have significantly helped reduce total cost of ownership (TCO) for both corporate IT organizations and service providers.

Today, the buzz is about IP-based storage networking. Technologies such as iSCSI, iFCP, mFCP, and FCIP are spawning dozens of new companies and initiatives to address the next wave of storage networking. It is my position that we consider the various IP-based storage schemes and contemplate the idea of multiple protocols supporting different applications before we totally disregard Fibre Channel SANs.

Fibre Channel benefits

By now, most readers know that Fibre Channel provides a media access control (MAC) layer protocol over which any ISO level 2 layer protocol can run.

For most networking protocol stacks, the concept of a data packet or datagram maps directly onto the Fibre Channel sequence. The sequence is segmented into frames, transmitted across the Fibre Channel fabric, and then reassembled before delivery to the remote level 2 protocol. For example, IP, SCSI, and ATM all map onto the Fibre Channel physical layer protocol.

Figure 1: Once data reaches the storage router, the IP frames are de-encapsulated and raw SCSI commands are forwarded to the storage device(s) over Fibre Channel.
Click here to enlarge image

So one big benefit of Fibre Channel from a business point of view is its ability to switch the upper-layer SCSI protocol between sources and destinations over the Fibre Channel fabric. This process allows the former point-to-point SCSI bus to now be switched, enabling the benefits of improved uptime through multi-pathing and reduced cost through device sharing and centralization.

Two of the key components of Fibre Channel environments are host bus adapters (HBAs) and device drivers. In the past, lack of compatibility among device drivers, HBAs, switches, and storage devices was a cause for concern. Today, this is less of an issue as standards have solidified and interoperability "plugfests" have helped vendors iron out the kinks of end-to-end interoperability. However, even today, with new 2Gbps Fibre Channel solutions coming onto the market, interoperability challenges continue to exist.

IP-based storage benefits

IP-based storage promises to further reduce TCO. An IP-based SAN removes the requirement for Fibre Channel HBAs and Fibre Channel switches. IP proponents assert that Fibre Channel switch ports are two to three times as expensive as IP switch ports. Since no Fibre Channel HBA is required in an IP storage network, another expense can be avoided. IP networks are ubiquitous and have established standards for both quality of service and security. Finally-the argument goes-since IP networks are deployed and interconnected worldwide, the realization of global SANs is achievable.

First, let's look at iSCSI, one of the first IP-based storage initiatives. iSCSI enables the SCSI protocol to be switched and routed over IP networks. Today, the primary method for doing this is to use a storage router that connects to a Fibre Channel interface on one side and to an Ethernet network on the other. A storage router from Cisco, with one Fibre Channel port and one Gigabit Ethernet port (1x1), lists for $25,000. At the host end of the IP SAN, Cisco requires the installation of a specialized device driver. This driver appears to the host's volume manager as a locally attached SCSI disk. When the volume manager writes to the local disk, the driver accepts the operations, encapsulates the I/O, and re-directs it to the IP address of the storage router. Once the data reaches the storage router, the IP frames are de-encapsulated and raw SCSI commands are forwarded on to the storage device over Fibre Channel (see Figure 1).

This encapsulation technique works fairly well. However, there are a few problems.

  • The device driver is very CPU-intensive. In response to this issue, unique iSCSI HBAs are being developed that perform SCSI/IP encapsulation outboard from the host.
  • The storage router is expensive ($25,000 in the case of Cisco's 1x1 router). And what about latency and single points of failure?
  • A third potential problem is Ethernet. Let's just say that your mileage may vary using Ethernet versus Fibre Channel.

Finally, let's do some math. Say you have five hosts, each with two paths in a SAN, connected to a 16-port Fibre Channel switch with six Fibre Channel connections to the storage array. That means:

10 HBAs$2,000 each$20,000
One 16-port switch$30,000 each$30,000
Total cost $50,000

Assume that the IP SAN can run over an existing Ethernet network, requiring no more switch ports, and assume that you do not need any specialized iSCSI HBAs. You still need to spend the following:

Six Cisco
storage routers
$25,000 each$150,000

And the benefits are a shared storage array and multiple paths. But can't you already get that with Fibre Channel? And don't you end up using extra CPU cycles for iSCSI processing?

Choose the right tool for the job

It might sound like I am not a fan of IP SANs, but that is not the case. The real issue is choosing the right tool for the job and avoiding vendor hype. Actually, there are several very exciting things happening in IP SANs.

First, we are going to see a new breed of iSCSI storage array that does not require an expensive storage router. The array will have Gigabit Ethernet interfaces on the front-end and will speak iSCSI to host computers. The host computers may require new iSCSI HBAs, but the overall cost model will threaten Fibre Channel SANs because no Fibre Channel switch will be required. It is also likely that many of these new iSCSI arrays will use IDE disk drives, which are much less expensive than the Fibre Channel or SCSI alternatives.

Figure 2: If the local (source) site goes down, the remote (target) site takes over and operations continue non-stop.
Click here to enlarge image

The promise of IP SANs is geographically distributed network storage. Here's where technologies such as FCIP may come into play. FCIP enables Fibre Channel fabric services to be tunneled through the IP network. By combining FCIP with existing technologies, new functionality could enable more inexpensive data- protection schemes.

Consider technologies available today such as EMC's Symmetrix Remote Data Facility (SRDF), which enables a data volume to be mirrored between two geographi cally separate Symmetrix arrays. Support for SRDF using Fibre Channel over IP is currently available. With SRDF, a customer keeps copies of the volume in two locations and attaches servers to each Symmetrix box. If the local (source) site goes down, the remote (target) site takes over and operations can continue non-stop (see Figure 2).

SRDF is excellent technology, but it requires that the customer's server farm be replicated at each site. SRDF does not allow for the local Symmetrix to fail and for the local servers to fail-over to the Symmetrix at the remote site. The result is that many would-be customers do not take advantage of SRDF because of the cost of duplicate servers and application licenses.

Figure 3: In this setup, in the case of a failure the applications will remain up, and the need for a second server farm at the remote site is eliminated.
Click here to enlarge image

By deploying FCIP with SRDF, as well as EMC's PowerPath, this problem of a second server farm can be eliminated. PowerPath is server-based software that enables fail-over from a primary storage array to a secondary storage array. Simply define to PowerPath that the primary volume (source) is the local disk and that the remote disk (target) is the secondary disk.

Now you can run SRDF as usual, but you do not have to deploy a second server farm at the remote location. If the local Symmetrix fails, or the local volume gets blown away, PowerPath will fail the servers over to the secondary volume. By using FCIP-enabled storage routers, you could now tunnel the PowerPath I/O over the IP network to the remote storage array. While performance will be throttled by the bandwidth of the IP network, the applications will remain up and the need for a second server farm at the remote site is eliminated (see Figure 3).

Future roles for IP SANs

Over the next three to five years, we may witness the transformation of storage from hardware boxes directly attached to application servers to a globally distributed storage network-a "storage grid"-presented to consumers as a set of service levels that are bought and sold in a commodity exchange.

This notion of a storage grid cannot be delivered unless storage technologies are interconnected using high-speed bandwidth and common protocols. Although we have many interesting file system schemes for globally distributed data, not all of the world's data is suited for file systems. So the storage grid must be a combination of file- and block-level data.

Today, we do not have a common data movement protocol to facilitate the exchange of block-level data between storage arrays from different vendors. Likewise, we have not solved the latency and coherency issues associated with globally networked storage.

Through the efforts of organizations such as the Internet Engineering Task Force and Storage Networking Industry Association, IP SANs will continue to improve and standards will be established such that a global storage network can be achieved. In the meantime, however, caveat emptor on the promise of iSCSI and other IP SANs.

Walt Hinton is chief architect and general manager at ManagedStorage International (www.managedstorage.com) in Broomfield, CO. He can be reached at walt.hinton@managedstorage.com.


IP-based storage protocols defined

There are several IP-based storage protocols under development, including the following:

  • FCIP provides a mechanism to "tunnel" Fibre Channel over IP-based networks. This enables the interconnection of Fibre Channel SANs, with TCP/IP used as the underlying transport to provide congestion control and in-order delivery of data. Standard Fibre Channel fabric services are used for switching and routing.
  • mFCP uses UDP rather than TCP as the transport protocol. Similar to FCIP, mFCP enables SANs to be interconnected via tunneling. While UDP is faster than TCP, the IP layer now is responsible for reliability and packet sequencing.
  • iFCP is a gateway-to-gateway protocol where TCP/IP switching and routing components replace the Fibre Channel fabric services. iFCP uses TCP for congestion control, error-detection, and recovery.
  • iSNS is a common discovery, naming, and resource management service for all of the IP storage protocols.
  • iSCSI is a transport protocol that operates on top of TCP, enabling the encapsulation of SCSI I/O over an IP network. iSCSI is a broader protocol than the others mentioned above and is explained in more detail in the article.

This article was originally published on November 01, 2001