In many cases, the best solution may be a combination of Fibre Channel and IP storage protocols such as iSCSI, iFCP, or FCIP.
By Tom Clark
Fibre Channel is the dominant protocol and transport for storage area networking (SAN). Being first to market with a high-performance solution for moving block data at gigabit speeds, Fibre Channel has been the enabling technology for a wide spectrum of shared storage solutions, including storage consolidation and streamlined backup operations. Over the past several years, however, new storage protocols have been developed to transport block data over mainstream IP networks. As in the old SAN-versus-NAS debates, users are now confronted with alternative approaches to shared storage and must determine which technologies will best suit their specific needs.
This article provides a brief overview of the currently available options for Fibre Channel and IP-based SANs, with recommendations for cases in which Fibre Channel, IP, or a combination of both should be used.
Fibre Channel strengths
Fibre Channel provides multi-gigabit speeds and transport protocols that are optimized for moving massive amounts of block storage data between hosts (initiators) and storage devices (targets).
Compared to other network architectures, Fibre Channel creates a flat, link-layer network that is analogous to bridged LANs. As multiple Fibre Channel switches are connected together, they become a single fabric. This design ensures maximum performance when storage data is sent from source to destination across multiple switches and imposes minimal protocol processing overhead. Fibre Channel switches, for example, use "cut-through" switching. The switch logic only needs to interpret the destination address to start processing the Fibre Channel data frame and send it on to its destination. In contrast, "store-and-forward" switches must buffer the entire frame or packet before routing can begin. In addition, Fibre Channel host bus adapters (HBAs) efficiently process block storage data at the adapter level, often imposing less than 10% CPU overhead on the host server.
The iFCP and FCIP protocols can be used to link geographically distant Fibre Channel SANs.
In data-center environments, Fibre Channel directors provide high port density and a non-blocking architecture for efficiently switching multiple concurrent storage transactions. Director-class Fibre Channel switches typically provide high-availability features such as hot code load and redundant processors. Departmental switches typically have 8 to 16 ports and may be linked to directors to provide fan-out from the data center to secondary departments or branches.
In terms of speed, Fibre Channel currently offers both 1Gbps and 2Gbps throughput, and initiatives are underway for 4Gbps and 10Gbps interfaces. To optimize throughput in a multi-switch configuration, vendors may provide the ability to aggregate inter-switch links and thus create fatter data pipes between switches. For less-demanding storage applications, Fibre Channel also provides an arbitrated-loop topology, which enables multiple storage devices and servers to share a common link, analogous to shared Token Ring segments.
IP storage protocols
Moving block storage data over IP networks has required the development of new IP storage protocols that can leverage existing Ethernet and IP network infrastructures. The three IP storage protocols currently available in products are Internet SCSI (iSCSI), Internet Fibre Channel Protocol (iFCP), and Fibre Channel over IP (FCIP). Each protocol provides a specific solution for IP-based SANs, giving users options for solving storage problems.
Both the Fibre Channel protocol and iSCSI encapsulate SCSI commands, status, and data in a serial transport. For Fibre Channel, the protocol wrapping is designed for a specific Fibre Channel transport, such as point-to-point, arbitrated loop, or switched fabric. For iSCSI, encapsulation of SCSI is designed for transport over any network that supports the IP protocol, including Ethernet, Packet over SONET (POS), frame relay, or ATM.
For end-to-end reliable transport, iSCSI uses TCP/IP protocols. If a packet is lost during transport due to network congestion or a routing error, the TCP layer will ensure packet recovery at the destination. Compared to Fibre Channel, TCP imposes additional protocol overhead on the host, and vendors have responded with TCP offload engines (TOEs) that move TCP processing to an adapter card.
iSCSI is designed for both initiators and targets, offering an alternative to Fibre Channel-based SANs. However, few vendors are marketing iSCSI storage targets, so iSCSI is sometimes positioned as a means to link additional servers to existing Fibre Channel storage. In this case, an iSCSI-to-Fibre Channel gateway is required to perform protocol conversion between IP and Fibre Channel.
iSCSI can be run as a device driver on top of a traditional Fast Ethernet or Gigabit Ethernet adapter card. However, this low-cost option incurs a penalty in terms of CPU utilization on the host system. TOE-enabled iSCSI adapters solve the overhead issue but add cost to each host.
iSCSI can be used to connect iSCSI servers to iSCSI storage targets, or iSCSI servers to Fibre Channel storage via a gateway. iSCSI is not typically used to connect Fibre Channel SANs across IP networks, however, since there is not a direct translation between all iSCSI and Fibre Channel commands. For Fibre Channel extension over IP, either iFCP or FCIP may be used.
iFCP is a gateway-to-gateway protocol for connecting Fibre Channel end devices or fabric switches over IP networks (LANs, MANs, or WANs). iFCP products thus have a dual personality in providing the capabilities of a Fibre Channel switch and Gigabit Ethernet/IP switch on a single platform. iFCP maps each storage conversation between Fibre Channel initiators and targets to pairs of unique IP addresses. This enables each transaction to be monitored across the IP network and enables full multi-point routing.
Like iSCSI, iFCP uses TCP to ensure data integrity across the IP network. TCP processing, however, is transparent to the Fibre Channel end systems and imposes no performance penalty on servers. The challenge for iFCP design is to provide Fibre Channel-to-iFCP protocol conversion at wire speed as well as efficient TCP processing.
An iFCP gateway may attach directly to a Fibre Channel host or storage device, or connect to an existing Fibre Channel fabric via E_Ports. In the latter case, the iFCP protocol differs from conventional switch-to-switch E_Port connections, since Fibre Channel fabric-building protocols are not passed between gateways. Fibre Channel fabrics that are linked by iFCP gateways thus maintain their autonomy, and fabric reconfiguration events and state change notification broadcasts are blocked. This SAN routing capability offers fault isolation between SAN islands, while enabling authorized access between SAN islands for designated storage devices and hosts.
Similar to the iFCP protocol, FCIP is designed to link geographically distant Fibre Channel SANs over common IP network equipment. Unlike iFCP, FCIP relies on E_Port connection to existing Fibre Channel switches and tunnels all Fibre Channel-originated traffic from one location to another. Since this includes fabric-building protocols, two geographically separate SANs linked by FCIP become a single logical Fibre Channel fabric. This stretched E_Port connection extends the Fibre Channel fabric over distance and is commonly referred to as SAN extension, as distinguished from iFCP's SAN routing or internetworking.
FCIP is a straightforward means to link Fibre Channel SANs over IP, and from the standpoint of the Fibre Channel switches at each site it appears as a local fiber link. FCIP has been adopted by many vendors as a convenient means to pass Fibre Channel over IP infrastructures.
Fibre Channel or IP, or both?
The first point in the Fibre Channel or IP decision tree should branch on the user's application requirements. High-performance applications are better served with Fibre Channel's 2Gbps performance compared to Gigabit Ethernet's 1Gbps speed. For data-center environments in particular, Fibre Channel's ability to provide 2Gbps attachment among servers, switches, and storage devices is the optimum complement to transaction-intensive, high-performance business applications. In addition, some vertical market applications such as high-definition video (approximately 130MBps per stream) mandate a full 2Gbps link.
Although not all business applications require multi-gigabit bandwidth, storage managers who are designing SANs for both current and future needs may opt for Fibre Channel to accommodate new, more-demanding applications in the future.
For storage over distance, a combination of Fibre Channel and IP storage protocols is often more cost-effective than Fibre Channel alone. Using dense wavelength-division multiplexing (DWDM) technology, native Fibre Channel can be extended through optical cable over metropolitan distances. However, DWDM or Fibre Channel over switched optical networks requires special equipment that may be beyond the budget limitations of many companies.
Using Gigabit Ethernet services in metro or regional area, or IP networking over wide areas, is typically more economical than using dedicated dark fiber links.
For disaster recovery or remote tape consolidation, users may elect to use FCIP SAN extension, or, for fault isolation, iFCP SAN routing. It is possible to span thousands of miles between Fibre Channel installations and, depending on vendor implementation, achieve full utilization of multi-gigabit IP links. In addition, use of IP storage products provides users with the flexibility to use lower link speeds between SAN sites or to leverage bandwidth management to share storage and messaging traffic on common links.
Combining Fibre Channel and IP SAN protocols is also beneficial for expanding SAN connectivity to hosts. High-performance servers may be connected via Fibre Channel, while less-demanding servers may be connected via iSCSI and iSCSI-to-
Fibre Channel gateways. The ability to attach second-tier and third-tier Wintel servers to existing Fibre Channel SANs through iSCSI enables users to amortize the cost of their SAN investment over much larger populations of devices while extending the benefits of shared storage and tape backup to less-critical platforms. iSCSI connectivity to Fibre Channel storage also brings blade servers into the SAN equation, thus providing new options for users to scale their server platforms to specific application requirements.
For low-end applications, companies can take advantage of new, lower-cost departmental Fibre Channel switches and modular Fibre Channel storage, or implement iSCSI hosts and iSCSI storage. Some currently available products also enable iSCSI-connected hosts to access conventional SCSI disk targets as well as Fibre Channel targets. An iSCSI SAN does not necessarily exclude Fibre Channel, since Fibre Channel hosts or storage devices may be introduced over time to accommodate additional higher-performance applications.
Having overcome the initial friction between Fibre Channel and IP storage initiatives, these technologies are now fully complementary. From an end-user standpoint, technology selection is driven by application requirements, both in terms of performance and flexibility in deployment. The convergence of Fibre Channel and IP storage technologies is now enabling users to select the best options and to satisfy the storage needs of a much wider spectrum of data-center and remote applications.
Tom Clark is a technical marketing and SAN evangelist at McData (www.mcdata.com) and the author of the books Designing Storage Area Networks Second Edition and IP SANs (Addison-Wesley).