Extending SANs to IP networks

Fibre Channel over IP is an efficient way to link geographically dispersed storage area networks, and iSCSI is waiting in the wings.


For many companies, managing the business means managing the data. Ideally, this data should be instantly and reliably accessible from the many systems in use. Backup downtime, slow recovery from failure, and lost data are simply unacceptable.

Storage area networks (SANs) are a key to achieving these business goals. Compared to traditional direct-attached storage, SANs offer better data administration, greater performance, higher availability, and less-obtrusive data backup and recovery.

Yet SANs are both a solution and problem for many enterprises. Existing technology requires that SANs be set up using an overlay network. In addition, a storage network completely separate from the core Internet Protocol (IP) client/server network adds administrative and capital costs that enterprises and service providers are ill-equipped to support.

Historically, SANs have been based on Fibre Channel technology, topologies, and protocols. Fibre Channel uniquely meets the technical requirements for storage communications-for instance, requirements for high reliability, minimal latency, and processing efficiency-that could not be met by standard IP and Ethernet-based data networks.

Since Fibre Channel SANs are not IP-based, mapping and extension to WANs are not straightforward so SANs are usually implemented as islands. While SAN islands are superior to traditional direct-attached storage, they represent only the first step toward a fully networked storage infrastructure.

Business trends are driving companies toward storage network integration and away from using SANs as distinct network islands. When SANs are extended over long distances, enterprises can secure important gains by integrating SANs into the universal IP backbone WAN.

This article focuses on one solution-Fibre Channel over IP-that promises to preserve the investment enterprises have made in Fibre Channel SANs, while adding the benefits of wide-area IP networking. The article also covers the applications that Fibre Channel over IP will make possible as companies adopt wide-area storage networking.

Delivery of the full promise of SANs depends on the ability to achieve both the benefits of Fibre Channel SANs and those of IP-based LANs/ WANs. Unless an enterprise or service provider wishes to maintain two distinct optical-based WANs-one based on IP and the second on simply extending Fibre Channel traffic over completely separate optical facilities (termed Fibre Channel Extension)-the Fibre Channel SAN must somehow be integrated into the IP backbone.

IP-based SANs

Ideally, storage over WANs should combine the high performance and low latency delivered by Fibre Channel SANs with the cost effectiveness and usability of the IP WAN. Such a solution would allow WAN links to be shared between storage and normal data traffic, enable all WAN traffic to be managed through generally available IP management systems, allow multi-point networking, and, where ultra-high speed is not critical, make it possible to use the Internet backbone.

Two complementary approaches to delivery of SANs in conjunction with IP networks are emerging in the storage industry: Fibre Channel over IP, and iSCSI. These approaches offer two distinct, yet compatible, ways to leverage the high-speed IP backbone network for SAN traffic.

A Fibre Channel-to-IP interface can be integrated into optical edge routers to connect Fibre Channel SANs over IP WANs.
Click here to enlarge image

Fibre Channel over IP supports existing Fibre Channel-based SANs, mapping SAN traffic into the IP-based backbone network for SAN-to-SAN communications.

In contrast, iSCSI proposes replacing the entire Fibre Channel infrastructure with an IP network, relying on Ethernet for LAN-and IP-over-optics for WAN-storage networking (see "The promises of iSCSI," p. 38).

Each approach offers advantages and disadvantages. Companies should base their storage decisions on the timing of their requirement, their existing base of SAN equipment and software, and their network administration cost model.

Fibre Channel over IP

Fibre Channel over IP integrates Fibre Channel SAN islands into the IP backbone WAN. It supports existing hardware and software investment in Fibre Channel SANs, while increasing the value of these SANs by allowing all SAN-connected data to be accessed over the IP backbone.

Fibre Channel over IP accomplishes this goal by introducing a new network interface element between the Fibre Channel SAN and the IP network. This network interface element encapsulates Fibre Channel frames into IP packets and maps Fibre Channel fabric domains to IP addresses. The figure shows an implementation in which the Fibre Channel-to-IP interface is integrated into an optical edge router.

In this way, SANs can be extended across the IP network, yet the Fibre Channel fabric and individual servers, storage devices, and software need not be altered in any way. This approach can be executed more quickly than iSCSI, which relies on equipping both server and storage system with iSCSI components.

Fibre Channel over IP is ideal for companies that have already deployed Fibre Channel SANs. It is also appropriate for companies that have not yet deployed a SAN but have an immediate need for SAN benefits, including those that can only be realized when SANs are extended over the WAN.

As with the iSCSI solution, Fibre Channel over IP provides access to the IP backbone mesh, which means users can combine storage traffic with other WAN traffic for a more cost-effective solution. For instance, it is normally cheaper to purchase an OC-48 pipe than it is to buy multiple OC-12 pipes for different applications. This approach also allows storage traffic to use more bandwidth during low-traffic periods. The shared WAN link can be used for Web traffic during the business day, and for backup traffic during after-business hours.

Because IP traffic passes through routers, a single Fibre Channel-over-IP site can be connected to any number of other Fibre Channel-over-IP sites using a single link. Rather than requiring dedicated links from each branch office, for example, a single high-speed link from each site to the backbone network is sufficient to allow communications among all of the offices. All of the management tools acquired to manage IP WAN traffic automatically apply to the Fibre Channel-over-IP network: There's no need for a special management infrastructure for channel extension.

Applications for FC over IP

There are several basic applications for Fibre Channel over IP:

  • Storage networking within a single campus;
  • Extended area data protection (to ensure business continuity during system backup and administration, and recovery from disaster);
  • Content distribution (for high-bandwidth information such as sound and video files); and
  • Extended data sharing and virtual storage (an extended area SAN, with data appearing transparently to the application wherever that data may be physically stored. Although this application offers the most benefits, it also requires the highest speed and lowest latency communications.)

Storage networking within a campus

Fibre Channel over IP within a single campus can leverage existing high-speed network links and also allow use of IP management and administrative tools. As native Fibre Channel can extend across most campus distances, enterprises may want to use network engineering analysis to determine whether native Fibre Channel or Fibre Channel over IP is the most effective means of extending SAN traffic. In general, if a large number of SAN islands are to be linked together, Fibre Channel over IP is likely to offer a cost-effective solution even within the campus.

Properly configured and depending on the vendor, a Fibre Channel-over-IP interface may also permit communications between the Fibre Channel-based SAN and an iSCSI SAN implemented over the local Ethernet network. Deploying Fibre Channel over IP as a gateway between Fibre Channel SANs and iSCSI SANs allows all hosts to see all permitted data, regardless of whether the hosts or storage devices are attached to the Fibre Channel or Ethernet physical network.

Data protection

For many companies, the first implementation of Fibre Channel over IP will be for data protection. Data protection does not require millisecond response times, but it does need reliable high bandwidth and a manageable network infrastructure.

Backup is a "killer application" for existing SAN islands. SAN-based backup removes backup traffic from the LAN and, in some cases, from the servers, allows shared use of high-speed tape libraries and lets administrators provide more- centralized service.

Local storage of tapes or local tapeless backup systems, however, do not protect against a catastrophic site failure, and physically moving tapes to an off-site vault imposes its own costs:

  • Delays in moving tapes to and from an archival location;
  • Unreliability of transportation systems during periods of environmental hazards such as bad weather, earthquake, or flood;
  • Inability to use tapeless backup systems where remote storage is desired; and
  • Costs of operating a system where tapes must be physically handled, transported, and archived.

Fibre Channel over IP allows enterprises or storage service providers (SSPs) to provide off-site backup storage that can be instantly accessed for backup and restore. To the SAN backup software system, the remote tape or tapeless backup system looks as if it is attached to the SAN. Exactly the same processes are used as for local backup, but the backup data is transparently transported to the remote archival location via the WAN.

If serverless backup is available, it can be used in this context, reducing the impact of backup on customers. If a failure occurs and restoral is required, the administrator again can access the backup data using conventional SAN software techniques.

In addition to scheduled data backup for disaster-recovery, Fibre Channel over IP also facilitates remote real-time data replication and mirroring. This offers the ability to produce duplicate copies of transaction data in remote locations for data mining, analysis, and other business processes, as well as protecting the live production data for business continuity.

Content distribution

The idea behind content distribution is to bring content closer to the edge of the network so people can get information as fast as possible, with minimal latency. Distribution and caching also make more-efficient use of network resources. It's an ideal application for Fibre Chan-nel over IP.

The accelerated Internet will distribute high-bandwidth content to caches near where people want it, refreshing them with new material as needed. After first contacting a central Web location, users will be automatically redirected to the content cache at a nearby service node, which will provide frequently requested content. The entire process will be invisible to users.

This simple concept conceals several technically tough issues. For example, some content changes moment by moment, so the content must be continuously refreshed. Tomorrow's content will be much more dynamic and complex, with much greater use of streaming video and audio.

Distributing this high-bandwidth content from central sites to remote nodes will require reliability, speed, and cost- efficiency. Fibre Channel over IP can provide the right capabilities for this emerging application. Content distribution networks, for example, would be able to move content from storage devices at central hosting sites directly to storage devices throughout the network, without sending the data as file transfers intermediated by servers.

Extended data sharing

The ultimate dream of storage networking is universal connectivity between servers and storage, where storage is available anytime and anywhere, without regard to location or distance. Fibre Channel over IP cannot overcome the basic speed- of-light latency issues that make it difficult for a server to perform high-speed transaction processing if it is located around the world from its storage system. But it does allow the SAN to extend across geographical distances that are significantly larger than the Fibre Channel specifications allow.

Distances of up to several hundred kilometers introduce relatively little speed-of-light latency. Enterprises with multiple data centers located within a multi-state region, for example, may use Fibre Channel over IP to build extended SANs and provide SAN benefits to all servers. This application will generally be combined with a data protection environment.

As shown in the figure, servers in any location may access data from any other location on a native SCSI/Fibre Channel basis. Keep in mind that data sharing, especially across heterogeneous systems, is a work in progress within the SAN community. Services available via Fibre Channel over IP are identical to those available from a Fibre Channel SAN. If the SAN solution does not provide heterogeneous data access within the local SAN, Fibre Channel over IP will not resolve this problem.

Storage service providers (SSPs) hope to deliver value to enterprises by managing storage, backup, replication, and restoral-a concept often referred to as "virtual storage." Fibre Channel over IP provides an economical means for SSPs to offer services ranging from data protection to full data storage. Servers located on the enterprise site can logically view storage systems within the service-provider site as if they were local disks. The SSP can transparently monitor disk activity, assign additional disk capacity when an application demands it, and purchase disks as a pool of resources rather than as required for any single enterprise or application.


Both Fibre Channel over IP and iSCSI will be key elements in delivering high-availability, cost-effective storage solutions. Fibre Channel over IP can be implemented in the near term, allowing isolated islands of SANs to connect, communicate, and share resources.

Fibre Channel over IP extends the Fibre Channel installed base and technology. It bridges existing Fibre Channel SANs to the IP backbone and will also bridge existing Fibre Channel SANs to iSCSI. In addition, Fibre Channel over IP allows existing IP management tools to see and administer SAN traffic that passes over the WAN.

Fibre Channel over IP is likely to be used first to provide data protection, without the costs and failures associated with physical movement of tape media. It also becomes a key part of extending the SAN across moderate geographical distances-up to several hundred kilometers-for integrating SANs across multiple regional data centers.

Fibre Channel over IP is likely to become a critical tool for storage service providers, enabling them to provide high-value services to large enterprises. Extended SAN services over the IP network will add a critical step up the value chain for these providers.

Rob Preece is senior product manager at Lucent Technologies Optical Area Networks (www.lucent.com) in Richardson, TX.

The promises of iSCSI

iSCSI offers the promise of complete consolidation of SANs into a seamless Internet Protocol-based network and is a complementary approach to that offered by Fibre Channel over IP. Fibre Channel supporters offer four reasons why Fibre Channel technology, instead of more standard IP-based LAN/WAN technologies, should be used for storage networking today:

  • Fibre Channel currently supports higher speeds;
  • LAN traffic involves variable delays, potential congestion, and packet loss, which is unacceptable in storage systems;
  • TCP and IP packet processing is a high-overhead operation that absorbs system resources and increases latency; and
  • Fibre Channel storage solutions are readily available today (e.g., disks, storage subsystems, and host bus adapters), but there are no existing iSCSI equivalents.

iSCSI supporters believe that these limitations can be overcome.

Ethernet technology now offers line rates that are as high as those associated with standard Fibre Channel gigabit rate, although the Fibre Channel community is expected to roll out 2Gbps connectivity this year and will be migrating to 10Gbps over the next several years. Switched Ethernet, now the dominant form of Ethernet used in server farms and other high-demand systems, helps eliminate collisions and offers high reliability and low latency. Finally, the TCP and IP protocol processing can (theoretically) be performed in silicon. An iSCSI host bus adapter equipped with a TCP/IP protocol chip could eliminate significant overhead from the server and reduce server latency. With these technology advances, iSCSI could potentially be as fast as or faster than Fibre Channel. At that time, the storage industry is likely to make iSCSI disks, tapes, and storage systems available.

iSCSI will be able to consolidate SANs into an IP-based network and can be complementary to Fibre Channel over IP.
Click here to enlarge image

When technical issues are resolved, iSCSI promises a number of advantages:

  • Ability to use a single shared WAN for both data and storage traffic;
  • Existing IP switching and router networks can be used with no changes. (Note that the low latencies and high performance required for storage traffic will call for high-speed links. Sending storage data over the Internet, for example, is likely to be an inefficient approach. However, high-performance optical links will offer high performance even over long distances.)
  • Storage systems attach directly to the relatively inexpensive IP switch, which can be completely ignorant of the traffic flowing over it; and
  • Finally, iSCSI is multi-point capable, taking advantage of the IP-based network's routing capabilities. iSCSI almost automatically delivers reliable mesh-based networking for any-to-any connectivity.

This article was originally published on February 01, 2001