The production studio uses storage area networks for sharing hundreds of terabytes of digital content, and Serial ATA-based disk arrays to keep costs down.
By Michele Hope
In Hollywood, pushing the envelope of what's been done before can bring high praise from critics and industry peers. According to EFILM LLC, a leading digital intermediate film company and subsidiary owned by Pana-vision and Deluxe Laboratories, it can also bring you almost more business than you can handle.
That's the double-edged sword of success EFILM has had to deal with since it created the first 100% digitally mastered feature-length film in 2001, Paramount Pictures' When We Were Soldiers, directed by Mel Gibson. The film, mastered in standard-definition, or 2K resolution format, soon brought other filmmakers clamoring to EFILM's door for the same digital intermediate treatment. According to Bob Eicholz, vice president of corporate development, EFILM now routinely juggles between 10 and 20 film projects at any given time.
Due in large part to the reduced cost of disk storage, EFILM was also able to achieve another digital mastery breakthrough with its recent work on Spider-Man 2, where it produced the world's first 4K-format, high-definition digitally mastered feature film. "This film has four times the data on the screen as a 2K image. What you see on the screen is much better than anything that you can get out of a normal film process," says Eicholz.
"It would have been cost-prohibitive to do a 4K movie three years ago," Eicholz explains. "At that time, a single terabyte of storage cost about $20,000. If you needed 4TB for a 4K movie, the cost was astronomical. That cost is now in the $2,000- to $4,000-per-terabyte range."
In an industry where one 2K movie frame can consume more than 12MB of storage space, it's common for a single copy of a standard definition, feature-length film to take up 2TB to 3TB of disk space.
"A 4K movie consumes four times that much," says Eicholz. "So you can see why we have so much storage capacity."
The push into 4K digital mastering led EFILM to acquire an additional 120TB of storage earlier this year, in the form of several TP9500 disk arrays from SGI. The arrays are based on Serial ATA (SATA) disk drives, which are considerably less expensive than Fibre Channel or high-end SCSI drives.
Pushing capacity to the limit
Eicholz and his 15-person team of "data-traffic cops" meet at least daily to determine how best to manage the flow of more than 200TB of data stored on the company's 19 SANs. The SANs include SGI's InfiniteStorage SAN servers, CXFS clustered file system, and a mix of TP9400 and TP9500 disk arrays to manage and share data with users throughout the organization.
Eicholz estimates that when EFILM first begins work on a 2K film, the footage can take up about 3TB of disk. After three weeks, however, this capacity can quickly expand to more than 10TB as they add multiple versions of scenes, lower (1K) resolution proxies used for color-correction work, and other video versions. By the time the project is finished, the film's storage requirements usually drop back down to about 5TB. The data is then archived to LTO2 tape via the use of Veritas' NetBackup software and a 200-cartridge tape library from Overland Storage. "The only thing that we keep in our online storage is images that we're actively using," Eicholz explains. "Once we make a permanent tape archive for a film, we delete the files from the system."
In addition to the 200TB of data stored on 19 SANs, EFILM has more than 20TB of local storage for color-correction work, such as that being performed on the Hawaiian ocean waves and sky for the film Blue Crush.
Over the next two to three years, Eicholz expects 4K, high-definition, digital film to replace 2K as the industry standard used for feature-length films. To accommodate this, he expects EFILM's storage requirements to grow to as much as 800TB. He is looking forward to new generations of denser Serial ATA drives coming to market. While he admits that low-cost SATA drives are slower than other types of disk drives, his overall experience with them has been positive. "Although there was a batch of bad drives early on, for the most part they've been a godsend," he says. "They're fast enough for what we're doing. They're on the SAN, so we're not using them to play back in real time."
During its life cycle, data from a single movie can move through three or four of EFILM's SANs. In addition to content on the SANs, the company also has between 20TB and 30TB of local storage distributed across four color-timing rooms. These are the rooms where cinematographers view projected, digital 1K "copies" of their movie images and work with colorists to digitally color- correct each film sequence, using tools in a proprietary version of Discrete Logic's ColorFront color grading software. Work performed in these rooms might include tasks such as correcting the color of someone's eyes or—in the case of Universal Pictures' surfing film Blue Crush—changing the color of Hawaiian ocean waves and sky for film shot on overcast days.
EFILM has one SGI Onyx 3400 or 3800 system with 16 processors and 5TB of direct-attached storage in each color-coding bay. At about the same time the film is being scanned into EFILM's systems, Eicholz' team uses SGI's CXFS shared file system software to rapidly transfer 1K copies of each frame from the SAN to local storage in one of the color-timing rooms. He notes that in some cases they are able to transmit the data from the SAN at a rate of more than 600MBps.
Speed the workflow
Eicholz realized early on that they'd have to limit the amount of large-scale data transfers between the color-coding bays and the SAN. Utilizing local storage in each room to implement color correction was key. "If they were all to try and play back 2K files off our SAN, each room would require a data-transmission rate of 300MBps to 400MBps. No network could handle that bandwidth," he says. "Using low-resolution copies, we still need approximately 100MBps for transmission of 1K files from the local disk to the local computer. For 2K files, it requires a minimum of 200MBps to 250MBps. If you go below that it gets choppy."
EFILM's storage-related choices have also been influenced by the need to make film data available to as many users as possible—when and where they need it.
Using SGI's CXFS shared file system to share files among multiple users is one way they've managed to reduce the need for large data copies and transfers. "CXFS allows a multi-operational system to access SAN files transparently, even though you're on Windows computers," says Eicholz. "Most of our users can access the entire 200TB, without moving from one place to another."
Another way EFILM has engineered their systems to allow multiple users to access and manipulate the same film files at the same time is by designing their software with an effective use of metadata. For example, instead of impacting the original film images, cinematographers or colorists performing color-correction work actually end up creating a "recipe" of sorts—the metadata that describes any changes that should be performed on the image after EFILM finishes working on it. "While the cinematographer is changing original colors, he's not actually changing the original scans," Eicholz explains. When EFILM performs a final render or produces the final digital master of the film, all of the metadata changes made along the way then become incorporated into the original film images.
Ethernet and 'sneakernet'
While EFILM has ironed out many of the workflow issues surrounding data access that tend to plague some production studios, Eicholz is the first to admit technology still falls short when it comes to getting the final, digital masters from place to place at a reasonable cost: "The issue that's not been resolved yet is how to effectively transmit that volume of data—in a full-length movie, or even a scene—across a city or across the country."
According to Eicholz, if EFILM had to send even a few minutes of film footage to, say, London it would take them all night to transmit the footage electronically. While the technology exists to transfer such large volume files at a faster rate, most studios—including EFILM—find the cost to do so prohibitive.
Instead, EFILM resorts to the old-fashioned data-transfer method commonly referred to as "sneakernet." "We put the film on disks and put them on an airplane. It's much faster and cheaper that way." EFILM often uses standard RAID-protected SCSI or SATA disk arrays to ensure data integrity during transport. "If it's really critical, we'll send two copies on two different disk arrays," says Eicholz.
Although the company's film data resides on its primary Fibre Channel SAN, Eicholz knows it pays to have a backup plan for everything. In the odd chance that the SAN goes down, EFILM falls back to its Gigabit Ethernet network to access key data. "At the end of the filmmaking life cycle, we can't have downtime or introduce delays at any time. So we try to have as much redundancy as we can." Disk arrays in the SANs are all RAID protected as well, a good practice since several of the hundreds of drives they have in operation tend to fail daily, according to Eicholz. "We're in a constant state of rebuilding somewhere."
Michele Hope is a freelance writer and owner of TheStorageWriter.com. She can be reached at firstname.lastname@example.org.
Video Storage Fast Facts
The world of feature films and video consists of two basic formats: standard definition (SD) and high definition (HD). Standard-definition footage is played back at 2k resolution, which displays 2048x1556 pixels on the screen. In contrast, high-definition footage is played back at 4k resolution, or 4096x3112 pixels. Additionally, the amount of data generated per film can vary based on the number of extra effects incorporated into a movie.
- Average storage size of a finished feature film:
- At 2k resolution = 2tb
- At 4k resolution = 8tb
- Data generated as work product for feature film:
- At 2k resolution = 10–100tb
- At 4k resolution = 40–400tb
- Average size of a single frame of film (one second of film consists of 24 frames):
- At 2k resolution = 13mb
- At 4k resolution = 50mb
- Typical bandwidth required for real-time film playback:
- At 2k resolution = 300mb/sec
- At 4k resolution = 1.2gb/sec
- Number of 2gb/sec Fibre Channel host bus adapters (HBAs) required for real-time film playback:
- At 2k resolution = 2 HBAs
- At 4k resolution = 6–8 HBAs
- Number of 2gb/sec host bus adapters (HBAs) required for real-time video playback:
- SD, 2k-resolution video = 1
- HD, 4k-resolution video = 1–2
- Average amount of active data managed by a large digital intermediate facility = 200tb
- Storage required for one hour of uncompressed video:
- SD, 2k-resolution video = 75–150gb
- HD, 4k-resolution video = 360–900gb
- Bandwidth for real-time playback of video:
- SD, 2k resolution = 21–42mb/sec
- HD, 4k resolution = 100–250mb/sec