Storage networking options for moving terabytes

Your choices boil down to Fibre Channel SANs, iSCSI SANs, NAS, and InfiniBand-all at a variety of performance and cost points.

By Mark Brownstein

It may seem like a question you would ask a first grade student: Is 10 better than four? In this case, the “10” is 10Gbps Ethernet, and the “four” is 4Gbps Fibre Channel. However, for data transport, raw (theoretical) speeds don’t necessarily mean much. Coupled with network throughput numbers are other variables, such as disk drive performance, software, and cost. Studios that need the utmost in storage network performance are looking for answers that make sense. The good news is that your options are expanding.

CTO, Pacific Title and Art Studio
Click here to enlarge image

Today, 4Gbps Fibre Channel SANs are increasingly becoming the technology of choice for production houses requiring high-speed data movement. Andy Tran, chief technology officer at Pacific Title and Art Studio, uses 4Gbps Fibre Channel to move “hundreds of terabytes around,” but he says that even 4Gbps Fibre Channel “is still not fast enough.”

In recent years, many studios have been satisfied with 2K resolution, which is adequate for standard definition (SD) video and some films, but high definition (HD) requires 4K resolution. Each frame recorded at 2K (2,048 x 1,556 pixels) occupies about 13MB of storage capacity, while a 4K (4,096 x 3,112 pixels) frame requires about 50MB. And the typical bandwidth required for real-time film playback at 2K is about 300MBps, while playback at 4K can require 1,200MBps, or 1.2GBps, of bandwidth (see figure).

Archiving films requires even more storage. For example, Pacific Title and Art is creating archives scanned at 6K resolution. At 6K (6,144 x 4,668 pixels), one frame takes about 115MB, and one second of film at 24 frames per second (fps) requires about 2.75GBps of throughput.

Click here to enlarge image

Actual production requires many times this amount of bandwidth and may generate many terabytes of data each day. Typically, frames move from workstation to workstation, and department to department, and intermediate versions of each frame are created and stored. With all of this data movement through a studio, a 2Gbps data stream could easily mushroom to 3x or 4x as great.

According to Pacific Title’s Tran, in a typical day his studio generates as much as 8TB of data. With the industry moving from 2K to 4K (and, in some cases, even 6K), the amount of data being worked on in many studios can be in the multi-terabyte range. Handling these huge amounts of data, and the tremendous amount of bandwidth needed to transport the data through a production facility, is a challenge that’s typically met through the use of 4Gbps Fibre Channel SANs-but other technologies are poised to challenge Fibre Channel’s supremacy.

4Gbps Fibre Channel

Outside of studio environments, mainstream adoption of 4Gbps Fibre Channel-the most recent version of the venerable SAN standard-is going slowly, but adoption is more rapid at digital content creation (DCC) facilities that crave the fastest speed possible.

“In the broadcast area, adoption of 4Gbps Fibre Channel is growing rapidly, with many studios standardizing on 4Gbps,” says John Howarth, director of storage marketing at SGI. Part of the reason for the rapid adoption is that 4Gbps Fibre Channel SAN components are priced about the same as 2Gbps components-providing twice the performance without a price premium.

Meeting performance requirements often requires multiple pipes. SGI’s InfiniteStorage TP9700 disk array supports up to eight 4Gbps Fibre Channel connections for a peak bandwidth of 400MBps per connection. SGI, which resells disk arrays from Engenio, is not alone in offering 4Gbps Fibre Channel interfaces. iQstor’s iQ2280 disk subsystem supports four 4Gbps Fibre Channel host ports and up to 15 500GB Serial ATA (SATA) disk drives for a total capacity of 7.5TB. Xyratex is also shipping 4Gbps Fibre Channel arrays, with up to 48 SATA drives (24TB) in a 4U enclosure. However, other high-speed storage networking alternatives are attracting attention.

10Gbps Ethernet

The next generation of the ubiquitous Ethernet-10Gbps Ethernet, or 10 Gigabit Ethernet-can be used for NAS or iSCSI-based IP SANs and has been receiving a lot of buzz. However, adoption has been slower than expected-in part due to high cost and backward-compatibility issues. “I don’t see the price of 10Gbps Ethernet being competitive until 2007 or 2008, with volume deployment in 2008,” predicts Steve Thompson, chief technology officer at Xyratex.

Because Ethernet is “lossy,” actual throughput will be less than 10Gbps-perhaps significantly less. “If you have large file transfers on an Ethernet fabric, you’ll start to have issues with packet loss and dropping out,” Thompson explains. “The solution is to create excess bandwidth in the core, but this means you’ll need more switches than you would with a more reliable protocol like Fibre Channel.”

Ciprico is one of only a handful of storage vendors that offers a 10Gbps NAS server, the DiMeda 10G, which is targeted at newsroom production and automation, play to air, cable network delivery, video on demand, and other applications. Ciprico claims sustained performance of more than 300MBps and a capacity of 12TB per system.

However, SGI’s Howarth notes that 10Gbps NAS may not make sense for many studios-at least not at this time. “The infrastructure for 10Gbps negates the reason for having NAS in the first place, because most companies with NAS are cost-conscious and/or don’t have the bandwidth requirements [to actually need 10Gbps],” he says.

Although 10Gbps Ethernet is available, it’s currently expensive to implement, although that will change over the next few years as the protocol is more widely adopted. “Studios will reap the benefits from the major investments that will be made by the corporate market and prices will come down, although it hasn’t happened as fast as I expected,” says Mike Anderson, Ciprico’s chief technology officer.

An emerging alternative to using NAS on 10Gbps Ethernet is to build an IP SAN based on 10Gbps Ethernet and the iSCSI protocol (which is also an alternative to 4Gbps Fibre Channel SANs). Adaptec, for example, last year formed an alliance with Server Engines LLC to create silicon for 10Gbps iSCSI.

However, this option may not have a strong place in studio environments, according to Jason Lo, CEO at iQstor. “I see [10Gbps iSCSI] for certain vertical applications and also for communications backbones, but for the majority of applications in studios 4Gbps Fibre Channel will be the workhorse for everyday use.”

TOEs and iSCSI

Because of the high CPU loads associated with TCP/IP protocol management, 10Gbps Ethernet is expected to extract a fairly high proportion of clock cycles. Although iQstor’s Lo doesn’t see 10Gbps iSCSI as a significant challenge to 4Gbps Fibre Channel, Kianoosh Nagshineh, president and CEO at Chelsio Communications, doesn’t agree.

That’s not surprising, because Chelsio makes 10Gbps Ethernet adapters (often referred to as TCP/IP offload engines, or TOEs, although they also offload iSCSI protocol processing). “There is a perfect storm of new technologies now: SAS and SATA on the back-end and TOE and iSCSI on the front-end, with 10Gbps Ethernet challenging Fibre Channel,” claims Nagshineh. “HD and high-speed video require 10Gbps to the client. The only reason studios went to Fibre Channel was that, up until now, it was the only high-bandwidth solution that was available. TOEs offload protocol processing so that studios can use the CPUs for rendering.”

Click here to enlarge image

However, TOEs add cost to the equation. “TOEs cost too much and are sort of a Band-Aid solution,” says SGI’s Howarth. “It’s cheaper to add CPUs than it is to buy TOEs.”


Perhaps the interface that’s received the least attention is the one that may eventually be most significant: InfiniBand. Currently used primarily as a server-to-server cluster interconnect, InfiniBand is making its way into the storage arena. For example, vendors such as Isilon Systems and Engenio (and its resellers such as SGI) sell disk arrays with InfiniBand options.

“One of the areas where InfiniBand is making inroads is in supercomputing, where people are moving very large amounts of data at very high speeds,” says Ciprico’s Anderson. “Studios processing digital film have a lot of similarities to the supercomputing and high-performance computing space. They have large amounts of data and lots of calculations and need a lot of bandwidth.” (Ciprico plans to demonstrate a prototype of an InfiniBand-based disk array at the upcoming NAB show, to be held April 22-27 in Las Vegas.)

Click here to enlarge image

Anderson points out that you can meet the need for speed with multiple 4Gbps Fibre Channel links, but that approach can increase the number of (relatively expensive) switches you’ll need. With InfiniBand, you can move huge amounts of data over one pipe.

“Almost all Linux clusters now use InfiniBand as an interconnect protocol,” says SGI’s Howarth, “and unlike Fibre Channel, when you plug storage into InfiniBand you can share the storage between machines. InfiniBand gives you a performance increase versus Fibre Channel, and it’s less expensive.” (Performance comparisons depend on a number of factors, including the number of “lanes” used in an InfiniBand implementation.)

For more information, see “Is InfiniBand poised for a comeback?” Info-Stor, February 2006, p. 1.


Studios’ choices in storage technologies do not end at the storage networking (SAN and NAS) layer. There are also a number of new disk-level interfaces that provide appealing options. In addition to low-cost, high-capacity SATA disk drives, Serial Attached SCSI (SAS) is another option that is just beginning to emerge. “SAS will [eventually] be the most likely disk interface for workstations,” says SGI’s Howarth. “The SAS cable length is longer than SCSI, and the performance is amazing.” SAS is a serial interface, as opposed to the parallel bus architecture used in older SCSI implementations.

The next couple of years will see considerable changes in the storage landscape. In studio environments, 4Gbps Fibre Channel SANs will continue to gain acceptance. Meanwhile, 10Gbps Ethernet will become much more affordable and will enable high-speed IP SANs based on the iSCSI protocol, as well as super-fast NAS.

SGI’s Howarth predicts that, “while 10Gbps products are just now coming to market, they will mature over the next two years. Both 10Gbps iSCSI and InfiniBand will challenge 4Gbps Fibre Channel.”

The storage networking technologies available today meet most studios’ existing requirements, but the continued move toward 4K resolution, exploding storage capacities, and increased bandwidth requirements will push leading-edge studios into exploring new technologies.

Mark Brownstein is a journalist specializing in high technology and storage. He can be reached at mbrownstein@brownstein.com.

This article was originally published on March 01, 2006