A review of FCIP, iSCSI, SoIP, iFCP, mFCP, and iSNS and a look at the strengths and weaknesses of IP storage.
BY JEFF WHITE AND BILL PELDZUS
Considerable discussion has centered on the state of IP storage, making it difficult sometimes to differentiate hype from reality. Written from an independent consulting perspective, this article focuses on IP storage markets and attempts to clear up some misconceptions.
Defining IP storage
IP storage can be generically defined as the transportation of block-level storage data over IP-based networks. This differs from network-attached storage (NAS) because NAS devices serve files, rather than blocks, and also communicate with more-complex protocols such as NFS and CIFS/SMB.
NAS devices and IP storage devices can be similar and/or overlap in capabilities, which can be confusing to our customers, because both architectures support storage over an IP network. For the sake of clarity, this article focuses on IP storage as it relates to block-level storage.
There are currently three protocols vying for support in the IP storage market: Fibre Channel over IP (FCIP), Internet SCSI (iSCSI), and Storage over IP (SoIP), which includes Internet Fibre Channel Protocol (iFCP) and Metro Fibre Channel Protocol (mFCP). Although the acronyms for these protocols can be confusing, the protocols themselves are distinct.
The FCIP protocol encapsulates Fibre Channel frames within TCP/IP packets and tunnels them across a TCP/IP network. This permits the interconnection of multiple, geographically separated Fibre Channel "islands" into a single unified fabric. Each island has a virtual E-port (switch-to-switch port) connection to other islands via an FCIP link. The switched fabric forwards the Fibre Channel frames across the virtual E-ports to the appropriate island. Encapsulation and restoration of the frames occur without any intervention of the fabric switches. Because FCIP acts transparently, issues of Fibre Channel time-outs due to distance may impact performance and/or the ability to sustain consistent communications.
Fibre Channel hardware is designed for very high reliability of bit-level transport through the network. The Fibre Channel protocol expects this high level of reliability. FCIP introduces a somewhat lower level of bit-level reliability inherent in a MAN or WAN environment. If a momentary interruption of network connectivity occurs between the FCIP gateways, the fabric may be split, which could cause each Fibre Channel island to reconfigure. Despite these issues, FCIP can be an excellent option to fulfill the business requirement of linking remote storage area networks (SANs) that are closer in geography (thus a more resilient link) but are still outside the distance limitations of Fibre Channel.
The iSCSI protocol dispenses with the Fibre Channel protocol layer in its entirety. Instead, the regular SCSI command set is mapped onto a new TCP/IP protocol that results in iSCSI LUNs and targets. These mappings on some devices are hard-coded and, therefore, there is no discovery or name server. This requires manual reconfiguration with each network change. However, it does avoid the previously discussed FCIP problems specific to intermittent link failure.
An iSCSI router maps iSCSI LUNs to Fibre Channel LUNs, and vice versa. The iSCSI router then unwraps or wraps SCSI packets in Fibre Channel or iSCSI. iSCSI uses TCP for transport of SCSI data through the IP network. The TCP protocol is a very complex protocol that does an excellent job of guaranteeing order and delivery of packets in the most efficient means possible. One tradeoff to this functionality is that it makes the TCP protocol very "heavy" in terms of computational complexity. High-speed TCP connections require large amounts of window space (or buffer memory) to operate efficiently. As an example, some iSCSI routers come with 512MB of RAM, with provisions for up to 2GB of RAM.
While TCP guarantees order and delivery of data, it may not use all of the bandwidth available at very high speeds. If TCP drops or receives a corrupted packet in a data stream, it assumes this was due to network congestion. TCP will then throttle back the speed of the connection substantially. Then it will slowly increase the speed, potentially taking several minutes to recover to full speed. The idea is for the TCP protocol to find the "sweet spot" for how much bandwidth is available between two hosts in order not to drop packets. This works well in WAN and LAN environments where many potential bandwidth choke points exist. In a switched Gigabit Ethernet SAN, congestion is less likely to occur; therefore, the benefits of TCP are somewhat reduced.
Today's Fibre Channel host bus adapters (HBAs) have a very high degree of onboard, integrated intelligence, usually a high-speed RISC processor, and at least 8MB of RAM. These HBAs offload much of the work required to operate efficiently on a Fibre Channel SAN, taking care of almost all layers of the Fibre Channel stack.
In contrast, a typical Gigabit Ethernet adapter generally has a very low level of intelligence. Most adapters only implement the Ethernet protocol and leave any higher-level protocols (i.e., IP) up to the host for processing. There are challenges for using iSCSI over a normal Gigabit Ethernet adapter. Large amounts of local CPU power and memory are needed to achieve high performance because of the TCP overhead. The iSCSI equivalent solution to a Fibre Channel HBA is sometimes referred to as a storage network interface card (SNIC). These boards are just starting to become available and are in the very early adoption stage.
Between the host and the storage device, a standard TCP/IP network could provide the sole network connectivity. But due to the potential for large volumes of storage data on such a network, most users may want to keep it separate from their primary LAN. Thus, much as a Fibre Channel SAN is separate from the rest of the network, it may be best to keep an Ethernet SAN similarly isolated.
The iFCP protocol operates at a higher level than iSCSI or FCIP. It has a Name Server, Internet Storage Name Service (iSNS), controls zoning, and other additional features. The package of iFCP, iSNS, and iSCSI is referred to as SoIP.
iFCP is similar to Fibre Channel, but instead of using the Fibre Channel protocol for transport, iFCP uses IP and TCP for layer 3 transport. iFCP protocol using UDP instead of TCP is known as mFCP (Metro Fibre Channel Protocol). The mFCP protocol enables the creation of a fabric of devices, much like Fibre Channel. An mFCP or iFCP switch would have both IP (usually Gigabit Ethernet) and Fibre Channel interfaces and implements a gateway-to-gateway architecture. This is accomplished by using iSNS, which provides a Worldwide Name (WWN)-to-IP address look-up table within each SAN fabric. When a Fibre Channel frame is sent to a device in a different SAN fabric, the frame is encapsulated and forwarded over the appropriate TCP/IP link to the SAN fabric that contains the destination device. The encapsulating wrapper is then stripped away by iFCP or mFCP, and the Fibre Channel frame delivered to the appropriate device.
mFCP in a LAN or SAN environment uses UDP for transport, which is fast and easy to implement, but unlike TCP, does not guarantee order or delivery of packets. Since Gigabit Ethernet uses the 8B/10B bit-level encoding taken from the Fibre Channel specifications, it has a low bit-error rate. Gigabit Ethernet also adds the IEEE 802.3x flow control standard. The ability to use flow control and low bit-error rates allows mFCP over UDP on a Gigabit Ethernet network to approach the reliability of a Fibre Channel network. Using UDP eliminates much of the protocol overhead and potential performance problems associated with TCP.
In cases where UDP would not work-such as over a longer-distance WAN link-iFCP can be used instead of mFCP. Using TCP would have the same issues that iSCSI has with TCP, but TCP allows mFCP to operate over lower speed and less reliable network links. iFCP and mFCP are less susceptible to FCIP reconfiguration problems due to links going up and down because it has its own Name Server and can wait a longer period of time before reconfiguring itself. SoIP is also robust in self-configuration and, arguably, easier to manage.
IP storage strengths and weaknesses
IP storage, with its many features and capabilities, has a lot of promise, and IP networks are well-understood and ubiquitous. Interoperability will be a less significant issue with IP networks as it was in the early days of Fibre Channel development. Additionally, there is a much higher availability of IP networking expertise compared to Fibre Channel.
IP also has the ability to transport over almost any medium, from Gigabit Ethernet to Fibre Channel to T1-even to an analog modem. This ability to work with almost any networking environment is very useful in a large enterprise.
In contrast, IP storage networking does not yet have the same installed base, level of reliability, and performance as Fibre Channel networks. With the upcoming 10Gbps Ethernet, this may become less of a factor, although Fibre Channel will also be upgrading to faster connection speeds. Reliability and performance are very critical to the operation of a SAN, and regardless of the decision whether to go with Fibre Channel, IP storage, or a combination of both, it is important to understand these technologies and how they fit into current infrastructures.
Finally, as in all storage networking technologies, we strongly suggest a detailed assessment along with a thorough testing of the proposed environment, with the mind-set of attempting to "break it." This will help users to better understand where the potential pitfalls are before implementing it in a production environment. This is even more important in complex, heterogeneous storage architectures.
In an upcoming article, we will outline an independent test environment of IP storage protocols, along with the interesting challenges and issues associated with the installation of these environments.
Jeff White is a senior storage engineer, and Bill Peldzus is storage marketing manager in Imation Corp.'s Storage Professional Services division (www.imation.com/storageconsulting) in Oakdale, MN.