Digital content creation studios have unique requirements when it comes to their storage infrastructures. Most shops, whether large or small, require the utmost in performance, capacity, and content sharing and collaboration–all at an affordable price. The following case studies illustrate how seven studios have solved their unique storage challenges:
Clustered storage for U2 3D film
In storage circles, hearing about companies moving from direct-attached storage (DAS) or file-based storage to a SAN is fairly common. But in the case of 3ality Digital Systems, it was a move in the opposite direction that would ultimately prove to offer the 500MBps required in the production of a 3D concert film.
Needing to store and manipulate approximately 100 hours of film footage shot in South America during U2’s “Vertigo” tour, 3ality Digital set out to produce the first digital, 3D concert film.
According to 3ality COO/CTO Howard Postley, postproduction work took about a year and turned out to be no easy undertaking from a storage perspective. It ultimately required almost one petabyte of clustered storage capacity from Isilon Systems, in the form of 80 IQ6000 and EX6000 node pairs (12TB per pair). Postproduction work also required Isilon’s IQ Accelerator extension nodes attached to the cluster via InfiniBand to help scale aggregate throughput to 10GBps. IQ Accelerator nodes offer 10Gbps Ethernet connections on the front-end that 3ality attached to its 10Gbps Ethernet switch. This switch, in turn, links the Isilon cluster to a variety of workstations.
The “U2 3D” film project required 3ality to switch from its prior SAN architecture to an Isilon OneFS clustered NAS file system that offered the production studio the option to manage and share files within a single volume, using a single global namespace.
3ality Digital’s production problems were somewhat unique. For example, artists working on the footage couldn’t rely on low-resolution proxies like most other studios do. To get perfect alignment of the 3D images so that viewers could watch the complete footage in 3D without eyestrain, 3ality artists had to work digitally in real-time, with two separate image streams–one for the left eye and one for the right. This ensured the images would be perfectly aligned at the same time and meant that artists had to work in an uncompressed environment, reading both streams in real-time. In some cases, scenes would have as many as five layers of 3D with which to work. This translated into 10 uncompressed streams coming out at the same time. “You can do the math on that. It would require about 4GBps, which was challenging,” says Postley.
After looking at large SANs from a variety of vendors, Postley concluded that most of them couldn’t provide the performance and manageability the studio needed. “Only a couple of the SANs could provide 500MBps performance to one workstation. Many of them could provide high aggregate bandwidth, but few could consolidate that bandwidth down to single workstations,” says Postley.
One issue Postley encountered was the common 64TB file system boundaries he found in most SANs. “You can buy a large amount of aggregate SAN storage, but the biggest SANs we saw are chunked up into 64TB boundaries. Based on how we work, any file system boundary creates a real problem for us,” Postley explains. Since moving to Isilon’s clustered storage configuration, 3ality no longer needs a full-time person to reorganize the SAN. “We’ve replaced our SAN performance with direct-attached Isilon storage. We now have all workstations attached via 10Gigabit Ethernet. Those who need 500MBps performance can have it, and we can scale up as high as we want.”
High-speed RAID slashes production time
What kind of storage infrastructure does it take to edit and deliver seven high-definition (HD) episodes for the Food Networks’ “Good Eats” show in fewer than 10 days? According to Walter Biscardi, owner of Atlanta-based Biscardi Creative Media, much comes down to the speed and reliability of the underlying storage.
Over the past three years, Biscardi and two other full-time staffers have been responsible for producing 75 broadcast HD masters and 150 standard-definition (SD) broadcast masters. They’ve also been the recipients of many awards for excellence in HD broadcast and postproduction.
Biscardi credits his latest storage configuration with dramatic boosts in his staff’s creative output. He estimates that with a slower storage system, it would have taken him about 18 to 20 days to deliver the seven shows he was ultimately able to deliver in fewer than 10 days. The underlying storage engine that drives high-speed reads-and-writes in two of Biscardi’s three Apple Macintosh Pro-based creative suites is an 8TB RaidPro SATA array from Maxx Entertainment Digital. The array is direct-attached to the Mac workstations via Atto Technology’s ExpressSAS R380 RAID adapters.
The ExpressSAS R380 adapters support SAS and/or SATA drives, x8 PCIe host interfaces, and eight external ports. For high-performance environments, such as Biscardi’s, the adapters leverage Atto’s Advanced Data Streaming (ADS) technology for high I/O throughput and reduced CPU utilization.
Biscardi had previously used SATA-based arrays from other vendors, with mixed results. Some vendors provided the requisite performance, but only in RAID-0 configurations–a factor that didn’t inspire much confidence in the case of drive failures.
Biscardi tested SATA arrays from three vendors: Sonnet Technologies, Dulce Systems, and Maxx Entertainment Digital. The results? “Maxx had the fastest performance with RAID 5, which was what I wanted. It also had the best price.” The Maxx RaidPro array clocked RAID-5 write speeds of 488MBps and read speeds of 429MBps with 1920×1080 10-bit RGB frame sizes. “The speed allows me to work in real-time without doing very much rendering,” says Biscardi.
Interestingly, two of the three RAID vendors tested were also using the same Atto adapters. Being familiar with Atto’s adapters from previously used Medea systems, Biscardi enlisted Atto’s advice during the testing process to help tune the systems for optimal performance in uncompressed HD mode.
According to David Martin, an Atto application engineer, one trick for high-performance, uncompressed HD environments is to set the interleave size to 1MB, which allows larger blocks of data to be sent to the array. For mostly standard definition work, a better interleave setting might be 512KB, says Martin. “The lower [interleave] setting would probably give you the best of both worlds, where it’s still fast enough to do some HD work if you need to.”
CNBC streamlines workflow
The average CNBC viewer probably doesn’t spend much time wondering about the type of back-end storage infrastructure it takes to support the rich, dynamic images and real-time financial and business information routinely displayed on-screen.
Yet, behind CNBC’s real-time stock ticker, its dynamic 3D financial charts, and 3D and 2D graphics and animations, there’s a sophisticated storage infrastructure. Part of the NBC Universal Cable business, the CNBC cable network may just be the biggest consumer of storage across the NBC Universal cable family. Gary Kanofsky, CNBC director of digital production and broadcast technology, is sure of it. “Far more than any other group in the NBC family, CNBC is the leader in terms of the amount of storage we use, the amount of graphics we produce, and the amount of data we shepherd on a daily basis,” says Kanofsky.
To explain, it might help to think of CNBC’s back-end storage as supporting three distinct areas: 1) ingest, storage, and manipulation of financial data; 2) graphics production of still images and animation; and 3) news video ingest and video production. Combined, these three areas support CNBC’s production of 14 hours per day of live programming and another three hours of taped original content.
In the first area, CNBC’s operations represent more of a sophisticated high-end data warehouse than what you’d typically expect from a TV channel. That’s a unique differentiator, says Kanofsky in discussing the evolution of CNBC’s $135 million teleproduction facility built four years ago in Englewood Cliffs, NJ. “CNBC, in addition to being a teleproduction facility, is also a data production facility where we store, process, and present financial market data, using state-of-the-art tools,” he says.
Real-time feeds from the financial markets are ingested and dynamically stored in an Oracle 10g Real Application Cluster (RAC) based on Hewlett-Packard cluster hardware running Linux and an underlying 200GB EMC disk array. Individual transactions are manipulated in real-time as they are ingested in order for CNBC to dynamically produce a current stock price ticker and rapid snapshot graphics for on-air display. Real-time data is also stored in the Oracle system where journalists can search and retrieve financial data from the past 20 years to quickly render 3D stock trend charts.
In the second area–graphics production–another underlying set of storage systems is at work. Here, graphics workflow has two main goals: the rapid production and reliable storage of many hundreds of small (approximately 1MB) “still graphics” each day (such as an over-the-shoulder graphic about a story the anchor is discussing), as well as the production of massive 3D Maya-based animations. As an example of the latter, production of a five-second, uncompressed animation is likely to consume roughly 500MB of storage space, according to Rich Tallmadge, a CNBC graphics engineer.
Needing a storage architecture that would streamline CNBC’s production of more complex Maya animations, allow for growth, and still not break the budget, CNBC opted for two Apple Xsan systems at 15TB each, one acting as primary storage and the other as a backup in the event of failure. Connecting each Xsan to a 10-node render farm and about 100 Apple Macintosh workstations are two QLogic SANbox 9000 Fibre Channel switches, which can be configured for 4Gbps or 10Gbps Fibre Channel connectivity.
Tallmadge says CNBC has a variety of switches from different vendors throughout the facility. The choice of QLogic for CNBC’s graphics workflow, however, was fairly straightforward. “When it came to graphics, the price point and budget we had, and the functionality and scalability we needed, we found QLogic’s switches to be the best fit for the project,” he says. Because the switches support up to 128 ports (or 256 ports in a dual-chassis configuration), Tallmadge says that scaling the system is easy.
In the third workflow area for storage–video ingest, video editing, and on-air playout–CNBC has adopted a three-tied storage infrastructure. A Thompson Grass Valley system holds the nearline storage tier, housing 4,000+ hours of SD DV25 video, at a capacity of about 120TB. Meanwhile, a NetApp FAS3000 storage system comprises the middle tier, and a Quantum Scalar 10k tape library serves as the final archiving repository. Middleware helps migrate the data from tier to tier based on whether it’s been used over a certain period of time–typically, five days.
In terms of video workflow, Kanofsky illustrates how the back-end infrastructure is needed to support CNBC’s dynamic workflow.
“You have this massive video production infrastructure All this inbound video, coming off satellite or fiber-optic [feeds] is encoded, stored, dynamically accessed in low-res. Proxy, in high-res at the editors’ console, edited and pushed out to air for playback, while the edited material is put back on the server for storage. So, when the community wants to access the edited or raw material, they can.”
VFX studio balances cost, I/O
Morphing from a visual effects shop that does SD TV work to a full-fledged feature film and HD TV visual effects studio takes a different mindset when it comes to storage. It also takes a different breed of storage altogether.
Peter Muyzers, chief operating officer at Vancouver-based Image Engine, explains the difference between storage for small studios just starting out compared to where Image Engine is today. “Storage isn’t an issue when you start off. You can literally buy SATA disks that you throw into a shelf in a RAID configuration and off you go.” That can work okay when there are only a few people needing access to the storage.
But, then try adding an HD production workflow and a lot more people to the mix. “That has a toll on the central storage system,” says Muyzers. “You can no longer rely on your home-brew storage. The requirements increase in terms of the demand. Users are demanding the same throughput, whether there’s one or fifty people accessing the storage system. The [controller] heads need to balance the load.”
Planning to move into HD film and TV work a few years back, Muyzers and his IT team set out to strike what they thought would be a good balance among storage performance, modularity for growth, and cost effectiveness. “Finding the balance is about how cheap we can go versus how much production we can do at the end of the day,” he says.
Image Engine worked with storage consultants to craft an RFP that would specify the studio’s unique requirements for storage. Among those was the need to maintain sustained throughput of four frames per second for playback to multiple artists’ desktops, concurrently, if need be. Each HD frame is 1920×1080 pixels and consumes about 13MB of disk space. Real-time playback requires 24 frames per second, for a combined data throughput rate of about 300MBps. But, Muyzers and team determined real-time playback for an aggregate number of simultaneous users wasn’t necessary. Instead, four to five frames-per-second throughput would be sufficient for the workflow of various 2D artists, editors, and compositors.
The studio ended up upgrading its existing 3TB NetApp FAS940 storage system it had been using for standard definition projects with another 26TB of storage in the form of a four-node Net-App FAS3050 cluster running the vendor’s Data ONTAP GX operating system. In terms of how well the new system delivers throughput, Image Engine IT lead Jason Navarro says he’s seen no slowdowns in performance, regardless of whether one, five, or fifty people are connected to the system.
The NetApp GX system also scores high marks from Muyzers for scalability. Allowing the studio to quickly add either more disks or more controllers independently of each other is a big plus, he says.
TV operator relies on tiered storage
About four years ago, Premier Media Group (PMG) chief engineer Michael Day knew the Australian television operator would have to do something different in its broadcast IT infrastructure to best accommodate PMG’s aggressive future plans for more channels beyond its original two FOX SPORTS channels. One of the first things to go would likely be the old-school, tape-based ingest process that had been responsible for storing and distributing sports-related video feeds from various sources for live playout or further editing.
Around that time, Day was briefed on plans to bring two additional channels online: FUEL TV and the HOW TO Channel. This seemed the perfect opportunity to craft a leaner, all-digital prototype to develop a streamlined, end-to-end operation for manipulating D10/MXF content.
According to Day, the prototype would be based on a non-linear editing (NLE) workflow and make use of a more scalable underlying architecture to help Premier Media Group cost-effectively support not just the two new stations, but also any others PMG might add. In terms of storage, Day and Tony Scanlan, PMG’s CTO, were adamant about needing a system that could scale easily. “We didn’t want to be shoehorned into a system that couldn’t expand,” says Scanlan.
The prototype has since evolved to include three sets of storage configurations connected to two editing production facilities. On the front-end, Quantel and Avid systems are used for ingesting live feeds from multiple sources and for editing. Day estimates these disk systems store about 40TB of data. Premier Media Group also uses SGI Altix XE ingest and transmission servers to help ingest pre-packaged video. All three ingest systems are connected to PMG’s storage backbone: a 4Gbps SGI InfiniteStorage SAN running SGI’s CXFS shared file system. Between the Altix servers and SGI SAN, PMG uses another 40TB of capacity.
The final–archival–tier of storage is a Sony PetaSite tape library with approximately 800TB of capacity. After files reside on the SAN for about five days, most are then moved to the PetaSite archive with the help of SGI’s Data Migration Facility (DMF) software.
The prototype has since grown to accommodate PMG’s current production roster of nine channels. How well has it worked? “We’ve been able to kick out nine channels, as opposed to two, in four years,” says Day.
PMG has learned plenty along the way. “You have to be clear on the workflows and some basics of the system, such as the file formats, you plan to use,” says Day. Scanlan also cautions to watch for potential file format interoperability issues. In terms of storage capacity and bandwidth, Day has also learned a few lessons. “Whatever bandwidth you anticipate, you’ll need more. It’s never enough. And the disks will always be full,” he says.
VOD requires high-performance SAN
With leading networks and studios striving to embrace multi-delivery channels and the new age of video-on-demand (VOD), it should come as no surprise that Microsoft has its own sizable hat in the VOD ring. Going beyond its initial roots, Microsoft has evolved its early Xbox community of gamers into 10 million+ subscribers to its Xbox LIVE Marketplace. The Xbox LIVE Marketplace is an HD VOD service that offers downloadable films, TV, and news content from more than 35 networks and studios–including MGM, Paramount, and the Disney-ABC Television Group.
Microsoft’s internal production and postproduction studio, Microsoft Studios, handles the back-end infrastructure required to make VOD work. According to Kurt Grubaugh, a senior engineer and IT operations manager at Microsoft Studios, one critical factor to the studio’s VOD success has been having enough storage muscle to handle all the streaming I/O coming from more than 35 studios and networks, not to mention the need to edit and encode each piece to ready it for live playout from Microsoft.com to various channels.
Grubaugh and his team ended up buying 170TB of Fibre Channel SAN storage from DataDirect Networks, in the form of the vendor’s DDN S2A 9550 storage system. DataDirect claims the system offers up to 3GBps of sustained throughput for both reads-and-writes.
Finding a storage system that would fit Microsoft Studios’ unique workflow was critical, says Grubaugh. “We wanted a system that would help us accomplish a workflow that could handle the type of HD files we get from our partners. They provide us the content, we ingest it, and re-encode it so it’s playable with our DRM solution for Xboxes, or from whatever platform we chose to make this offering available.”
Grubaugh knew such a workflow would require a very high-performance disk controller. “The fastest disk controller we could find was from DataDirect Networks. It met our needs in terms of price and performance,” he says.
Grubaugh also liked the fact that DataDirect’s disk system could integrate closely with the studio’s Quantum StorNext data management software, which is used extensively for virtual file management and file sharing across different operating systems, as well as HSM-style data movement off the SAN to slower disk systems or the studio’s LTO-4 tape system.
On the current Fibre Channel fabric, Grubaugh estimates the DDN system can offer throughput of approximately 1,000MBps to each server or work-station node.
According to Grubaugh’s calculations, the studio requires sustained, high-level throughput to about 40 nodes to ingest full-resolution high-definition footage.
How well has the DDN system worked? “We’ve been so successful with this project, we are now going back and re-analyzing our other 27TB,” says Grubaugh. “We’re looking to do a fork-lift upgrade soon and buy another 100TB [from DataDirect] for our production side.” He figures it won’t be long before the studio facility “tops out” at about 300TB of DDN storage.
Scalable SATA for content archive
When TV station KYW-TV in Philadelphia set out to build its new facility a few years back, it became something of a trailblazer. Over the course of about 10 months of hard-core engineering, it would transform itself into one of the first all-digital, file-based HD news gathering/editing facilities in the country. Such a transformation has since led to numerous awards, including an award from the Pennsylvania Society of Professional Engineers.
Richard Paleski, director of operations at KYW-TV, learned a few things about tiered storage along the way. His facility’s storage breakdown includes 35TB on a Grass Valley editing system, 7TB on an Apple Xsan, 3TB on a Nexsan SATABoy disk array (with three more on the way), and 116TB on a SpectraLogic T950 tape library.
A few years back, Paleski went to a storage conference with one thing in mind. “We wanted something in between our main news/edit systems and our tape library,” he says. After seeing Nexsan’s SATABoy in action, he made the decision to buy a few terabytes. “The system was easy to expand, and it’s [cost-effective] to build the SAN with SATA drives.”
Now, all file-based news content is managed by Front Porch Digital’s DIVArchive content management software. DIVArchive policy is set to store 30 days’ worth of news material online (in the Grass Valley and Apple Xsan systems) for instant recall. Once a day, data that’s ready to be moved then goes to the SATABoy disk array, where it is stored for another two weeks.
DIVArchive’s Storage Plan Manager also oversees the daily write of what Paleski indicates is “one long file” from the SATABoy disk system to the Spectra-Logic LTO-3 tape library. According to Paleski, this once-a-day stream from the SATABoy array is a much better option than forcing the tape’s robotic arm to work continuously writing the data to tape.
One factor influencing Paleski’s decision to go with Nexsan and SpectraLogic was the inherent scalability of the systems. With the SATABoy, he knew he could eventually upgrade the existing 500GB SATA drives to 1TB drives and add more drives on a modular basis. With the SpectraLogic system, Paleski figured he could always add more tape cartridges, and even extra cabinets, if needed. “Once it’s full, we can keep about two years of HD video on LTO-3 tape. We can also add another cabinet to double the storage, and even grow up to five cabinets. That will give us the ability to have 10 to 20 years of HD video stored online,” he says.
