Who`s using Fibre Channel and why?
Increased performance often takes a back seat to other advantages of the serial interface.
By Dave Simpson
Fibre Channel offers a number of potential benefits over alternative interconnects such as parallel SCSI, including faster throughput, fault tolerance via dual loops, a large number of devices per loop, and long connection distances. But before you take the plunge, look at how your peers have justified the cost of migrating.
At some sites, the main reason for implementing Fibre Channel is simply the need for speed. That was the case at Nathan Printing, in St. Cloud, MN. "We couldn`t get enough speed with the host-attached SCSI storage we previously had, not even with Wide Ultra2 SCSI," says Lyle Meyers, system coordinator at the printing company.
Meyers was getting 2.5MBps to 3MBps with SCSI, and is now achieving throughput rates as high as 20MBps with Fibre Channel, a 6.6x to 8x performance increase. Although Fibre Channel is rated at a theoretical maximum rate of 100MBps (1.062Gbps), your results will vary for a number of reasons. "A lot of it has to do with overhead," explains Meyers, "especially in an NT environment in which a lot of the data gets cached by the server operating system."
Other factors affecting actual Fibre Channel throughput include frame processing overhead, the burst capabilities of the host bus, system configuration, traffic characteristics, and topology (e.g., point-to-point, arbitrated loop, or switched fabric). In addition, in TCP/IP environments, if an unmodified protocol stack is used on the host system, the protocol stack overhead may become a limiting factor on performance. In any case, maximum Fibre Channel performance increases as frame size increases.
Nathan Printing has three Windows NT servers connected to a XIOtech RAID array via a relatively simple point-to-point Fibre Channel connection, as opposed to the more complex hub-based arbitrated loop or switch-based fabric topologies. XIOtech claims its 1TB Magnitude disk arrays can handle more than 50,000 I/Os per second and can be configured in clusters with as many as eight attached hosts.
Although the need for speed was critical at Nathan Printing, performance often isn`t the main reason for switching to Fibre Channel. "We didn`t really look at Fibre Channel as offering great performance leaps," says Joe Aultman, a storage architect with BellSouth, in Atlanta. "The main reasons we went with it were connectivity [number of devices per loop] and to get past the distance limitations of SCSI."
BellSouth started a massive consolidated storage initiative last year, and is in the process of consolidating from six data centers to two centers. At one facility, servers and storage devices are located one to three floors apart, which is well beyond the 12-meter cable limitations (without bus extenders) of SCSI. In contrast, Fibre Channel theoretically allows connection distances of as much as 10 kilometers with single-mode fiber-optic cable. With copper and coaxial wires, the distance is limited to 30 meters. And with multi-mode fiber-optic cable, connectivity distance is limited to 500 meters.
Take it slow
Like many sites, BellSouth is taking a slow and cautious approach to implementing a large Fibre Channel environment. The company started out with point-to-point direct connections between host bus adapters and EMC Symmetrix disk arrays. BellSouth later added hubs (Hewlett-Packard hubs for HP servers, and Vixel hubs for Sun servers) and will eventually add switches in a fabric configuration. The company is currently testing EMC`s Connectrix Fibre Channel switch, which EMC OEMs from McData. (A switched fabric topology allows multiple simultaneous transmissions between pairs of nodes connected to the fabric network. The fabric routes or switches data frames from source nodes to destination nodes.)
BellSouth`s huge storage environment includes Hewlett-Packard HP-UX servers equipped with HP`s Fibre Channel host bus adapters and Sun Solaris servers with Fibre Channel host bus adapters from Emulex and JNI (formerly Jaycor Networks). The servers are connected to eight EMC Symmetrix 3700 disk arrays in a RAID-1 configuration with 3TB of usable capacity per array (6TB total per array). The arrays have host-level Fibre Channel attachments and SCSI disk drives.
Sometimes, companies are able to take advantage of a number of Fibre Channel benefits. For example, at RemarQ (a provider of Web-based and Usenet discussion group services for corporate, ISP, and individual users), the benefits of moving to Fibre Channel were three-fold: throughput, devices per loop, and cost.
RemarQ (formerly called Supernews) pumps some 2,500 random I/Os per second per array through RAID-5 subsystems from DataDirect Networks. RemarQ has five DataDirect EV-1000 arrays and an EV-5000 array (more than 8TB in all, and growing at a rate of 3TB per quarter) connected via Fibre Channel to six Solaris Enterprise 4500 servers. The configuration consists of a Fibre Channel loop with hubs from Gadzoox and host bus adapters from JNI.
But throughput wasn`t the only reason RemarQ migrated from host-attached SCSI storage configurations. "Fibre Channel allows you to create much bigger RAID loops, and if one server fails another one can pick up," explains Rich Turnquist, director of operations at RemarQ. "With SCSI, you`re very limited in how many disks you can put on the bus." SCSI is limited to 16 devices per bus, whereas Fibre Channel is theoretically capable of supporting 126 nodes per loop in a Fibre Channel Arbitrated Loop (FC-AL) topology (although performance degrades significantly with high device counts).
As for cost, Fibre Channel is considerably more expensive than SCSI, but Turnquist says the cost of Fibre Channel was less than adding capacity via network-attached storage servers from Network Appliance, which RemarQ also uses.
Keep it simple
Glen Raven Mills, a textile manufacturer in Glen Raven, NC, deployed Fibre Channel in large part because it requires significantly fewer host controllers than SCSI. "The biggest advantage of Fibre Channel is that it requires so few controller cards in the host, which was getting to be a major problem for us with Fast and Wide SCSI controllers," says David Harrison, director of information services. Before converting to Fibre Channel, Glen Raven was maxing out the slots on its HP9000 servers.
The company`s Fibre Channel environment consists of an MTI Gladiator disk array (760GB, mostly RAID-5) on two multi-hosted Fibre Channel loops with two HP9000 HP-UX servers and HP`s Fibre Channel host bus adapters. The loop also includes a Fibre Channel hub from Gadzoox.
At Glen Raven Mills, speed was not an issue. In fact, Harrison says Fibre Channel`s aggregate bandwidth isn`t any better than the bandwidth he was getting with SCSI. The reason, he says, is that they were using so many SCSI host controllers and highly balanced I/O. "If you`ve got six Fast Wide SCSI controllers and your I/O is balanced, that`s 120MBps of bandwidth, but with Fibre Channel you only need one pipe and it`s a more flexible approach," says Harrison.
One of the key differences between SCSI and Fibre Channel is that, at the physical level, Fibre Channel is a serial interface and SCSI is a parallel interface. With a serial interface, data can be transferred over a single piece of medium (such as a wire or strand) at the fastest speed the medium is capable of supporting. On a parallel interface, data is transferred over multiple wires simultaneously.
Software-based file sharing
If you have the right software, another reason to implement Fibre Channel is for high-speed file sharing among heterogeneous host platforms. That was the main reason that Kmart, the large retailer in Troy, MI, made the move to Fibre Channel.
Kmart is a good example of the many graphics/video operations (including pre-press, video editing and visual imaging) that have been early adopters of Fibre Channel. At one facility, Kmart has four Windows NT workstations (running Adobe After Effects, Photoshop, Inscriber character generator, CG Extreme, VMP 4.0, and FeaturePak) and two Power Macintosh systems (one running Avid Composer 9000 and the other running Avid Audio Vision with ProTools).
According to Jim Lecours, chief engineer at Kmart, the main reasons for implementing a Fibre Channel network were the need for high throughput to transfer large files (e.g., downloading image files from the character generator into the Avid system for compositing) and for data sharing across Macintosh and NT platforms. Prior to implementing Fibre Channel, Kmart used "sneakernet" to transfer files.
In addition to the host platforms, Kmart`s hardware environment includes a Brocade Fibre Channel switch and a 10-slot JBOD ProTower disk array from Transoft Networks. (Kmart upgraded from a hub-based arbitrated loop configuration to a switch-based fabric last month.) The company uses host bus adapters from Emulex and Qlogic. With one time line, the Avid 9000 system runs at about 20MBps, although Lecours says with more time lines it will probably go up to 70MBps or more.
Kmart`s production group produces training videos and broadcast programming for Kmart`s in-store satellite network. Future projects include more multimedia and DVD projects for in-store point-of-purchase multimedia stations.
The real key to Kmart`s environment is Transoft`s FibreNet software, which allows heterogeneous file sharing in storage area network (SAN) environments. FibreNet supports any combination of Windows NT, Unix, and Macintosh platforms, and provides network file control functionality.
Other vendors that make software for file sharing in heterogeneous Fibre Channel-based environments include Mercury Computer Systems and MountainGate.
The potential benefits of Fibre Channel do not come without problems, however. For example, Kmart`s Lecours says that when they had a hub-based arbitrated loop environment, LIPs (Loop Initialization Primitives) caused some problems, in some cases requiring reboots of all their workstations.
A LIP is one of many primitives in the Fibre Channel specifications. (Other examples of primitives include idle characters and index tabs.) A LIP consists of a four-byte primitive sequence, which is a transmission word defined by the Fibre Channel spec that conveys control information between ports on a link. LIPs are issued across the Fibre Channel network when, for example, a device is added to the loop, at which point all of the connected devices have to re-initialize.
LIPs can cause problems because re-initialization is a complex, seven-step process that requires a lot of activity on the part of the connected nodes, according to Lee Roll, an applications analyst with Ancot, a manufacturer of Fibre Channel and SCSI test equipment. This can cause errors because device manufacturers don`t all interpret the Fibre Channel specifications in the same way. LIP problems are expected to decrease as vendors step up their interoperability testing efforts.
The LIP problem is common in many early Fibre Channel implementations, particularly arbitrated loop configurations with hubs. (An arbitrated loop topology links up to 126 ports on a single shared simplex media. Transmissions can only occur between a single pair of nodes at any given time. Arbitration is the scheme that determines which node has control of the loop.)
"The biggest problems we had with Fibre Channel have come from LIP problems on the loop," says Glen Raven Mills` Harrison. "It usually happens when, say, a cable breaks or a controller fails. It generates a LIP on the loop and a host might stop communicating with the array."
Harrison`s solution to the LIP problem is twofold. First, the company is upgrading to Gadzoox hubs with more intelligence, which enables the hubs to sense LIPs and disable ports. Second, Glen Raven Mills is upgrading to high-availability controller firmware in the MTI Gladiator array, which will allow the array to automatically switch from one controller to another if there is a LIP on one of the loops, and to take one of the loops offline.
The only other problem Glen Raven Mills had when it deployed Fibre Channel was that at first the HP servers couldn`t "see" the RAID LUNs. As a result, the company had to use a Sun workstation to initialize the LUNs. After that, however, initialization posed no problems.
Aside from the LIP problem, minor gotchas, and operating system patches, the biggest potential hurdle in implementing a Fibre Channel network is incompatibility--or non-interoperability--among products from different vendors. "Product incompatibility is one of our major concerns with Fibre Channel," says BellSouth`s Aultman, echoing the opinions of other early adopters, "so we`re taking a very cautious approach and are restricting the number of storage vendors that we deal with."
Among early adopters of Fibre Channel, the dominant advice on non-interoperability issues is to hold as few vendors responsible as possible. "We pretty much rely on EMC to ensure that interoperability and compatibility are addressed all the way to the HBAs. We even buy the Sun HBAs directly from EMC," says Aultman.
Non-interoperability between Fibre Channel products from different vendors is reminiscent of the early days of SCSI, but the problem can be more significant with Fibre Channel because, in addition to operating systems, servers, disk arrays, host bus adapters and controllers, some Fibre Channel environments also include hubs and/or switches. (For more information on Fibre Channel interoperability, see the Special Report in the January issue of InfoStor, p. 22.)
As a result, Aultman`s general advice to any company embarking on Fibre Channel deployment is: "It is absolutely critical that you form partnerships with a very limited set of storage and processor vendors."
Fibre Channel storage networks can be configured in three topologies: point-to-point, arbitrated loop, and switched fabric.