BY TOM COUGHLIN
Art requires science, and digital storage delivers key technologies needed for the next generation of media and entertainment production. As the resolution and depth of video content increases, the amount of content that must be retained for immediate production, as well as future use—plus that required for long-term content preservation—increases. This article explores these two themes: digital storage for production, and for long-term retention and preservation of professional video and audio assets.
Digital production is using ever-greater resolution due to the steadily lowering costs for capturing and editing this content. At the same time, stereoscopic production to support digital cinema, and even home 3D video demand, puts extra strains on the bandwidth and storage capacity requirements of video production.
The table “Media format requirements” (above) shows characteristics for several common uncompressed and compressed professional media formats. As shown, as the content resolution increases, the data rate and resulting capacity demand increases.
The large file sizes required for high-resolution images lead to increasing demand for high-capacity storage devices. Up to 1,600TB may be required for a complete digital movie production at 4K resolution, and some production (such as IMAX work) is moving to 8K resolution. Stereoscopic production doubles the required content capacity at any resolution.
Feature film resolutions are on an upward trend, particularly at the very high end of the market. Today, 2K resolution films are very common and 4K resolution films are becoming more common. For some sequences involving fast motion or low light, 6K resolution is sometimes used for content capture. The table “Feature film metrics” (below) shows some metrics for 2K and 4K feature films.
The figure “Post-production storage capacity” (next page) plots the annual demand in storage capacity for post-production. The chart breaks out storage capacity for direct-attached and network-attached non-linear editing (NLE) storage capacity from compositing and special effects capacity. File-based storage on a network is expected to increase over direct-attached storage over the next few years as bandwidth increases and costs go down for network storage. Overall, it is estimated by the IBC that 75% of all data created for the media and entertainment industry will be file-based data by 2011.
The 3D effect
Stereoscopic media production and distribution is being driven by its adoption by digital theaters to provide a unique experience for movie viewers. At the same time, stereoscopic content for home use is seen as a way to create a compelling next-generation viewing experience. Finished 3D content is about 1.5x the size of 2D content, whether in the theatre or the home. Production and capture requirements for stereoscopic video are at least double in storage capacity and bandwidth requirements, compared to those for conventional video production. Combining ever-increasing video resolution with stereoscopic images multiples the storage and bandwidth requirements considerably.
Stereoscopic capture and production drive data rates for digital video streams to Gbps levels. Gigabit Ethernet networks will be required for the emerging market of higher resolution and stereoscopic content. In addition, to improve storage ROI, there is a need for greater efficiency in the use of storage assets. Large system architectures that support multiple users and requirements on a single file system are more attractive to many users than a number of smaller storage systems.
Virtualization, which is common in many enterprise applications, is not common in the media and entertainment industry due its high-end proprietary requirements. Virtualization works best on commodity hardware rather than the constantly developing hardware used in media and entertainment. On the other hand, studios are constantly looking for more cost-effective ways to support the greater storage capacity and bandwidth required for content, production, delivery and preservation.
For example, 3ality Digital Systems is using scalable storage systems from Isilon to create 3D live-action content. The company created the first film entirely shot, edited, and shown in digital 3D, using more 3D cameras in production than ever before for a single film project. This will be a film of U2, from the group’s worldwide tour. The project required accessing and managing multiple terabytes of data that had to be accessed concurrently by more than 65 users in 3ality’s Burbank, CA facil- ity and another 25 developers in Germany. As another example, Lightstorm is creating a 36TB single file system, using Isilon’s storage systems for storing content for production of the 3D film Avatar.
Saving for tomorrow
Preserving digital content and converting historical analog content to digital form will be the single largest driver of digital storage capacity. Much of the storage for archiving will be on removable media, such as tapes and optical disks that can be put on a shelf or in a library until needed. Tape is the most popular archival media, although some studios prefer optical disks; however, there is an increasing use of hard disk drives and arrays for long-term storage, since access to the content is faster and the assets can be repurposed more quickly for other applications.
Digital conversion and preservation allows content to be available for research, use, and distribution. Future content businesses will be based on the availability of vast amounts of historical digital content, as well as new commercial and semi-professional content.
Many major digital conversion and preservation efforts are underway. Examples of analog content needing to be preserved include:
- The 100,000-hour CNN video tape library
- Stock material at major networks
- ABC: 1,037,000 films/tapes
- CBS: 1,045,000 tapes and more than 150,000,000 feet of film
- NBC: 600,000 film reels (currently estimated at 100,000,000 feet) and 1,600,000 videotapes
- Materials accumulated by major studios
- Disney: 6,500 TV programs on 80,000 reels and tapes
- Fox: 54,000 TV programs on 780,000 reels/tapes
- Paramount (Viacom): 8,000 TV programs on 1,200,000 reels/tapes
- Sony/Columbia: 35,000 TV programs on 600,000 reels/tapes
- Warner Brothers: 28,000 TV programs on 1,000,000 reels/tapes
The older film stocks and substrates are aging rapidly, making content preservation a high priority. The industry is still in the early stages of digital preservation (e.g., Turner Classic Movies has only converted about 15% of its total content—enough to fill two Sun/StorageTek 8500 tape libraries).
As the resolution of archived and preserved content increases, so will the digital storage required. For example, a single 70mm film frame scanned at 4K may require 50MB, or 200MB at 8K. At 24 frames per second (fps), an hour of content would need 86,400 frames or 4.32TB and 17.28TB for 4K and 8K, respectively.
The NBA started a project in 2004 with about 8TB of central storage using SGI’s DMF hierarchical storage system to mask the|archive and active storage tiers. This has grown to about 8 petabytes (1,000x growth in four years). The growth is due to growing resolution of new content as well as an aggressive program to digitize and preserve analog tape and film dating back to the 1940s. The content in this archive uses MPEG-4 video with 50Mbps and 100Mbps SD and HD video files. As of 2008, about 30% of new content was HD, with this number expected to increase to 90% this year. Keys to making the digital content useful are data reduction and management tools that aid in effective storage tiering, metadata parsing, and advanced visualization of the content.
NASCAR has 60,000 hours of footage that it plans to convert to all-digital format within a few years. The priority is converting 14,000 hours of HD content, which is estimated to take about 28 weeks. NASCAR is converting five days a week and two shifts per day using DataDirect Networks storage systems (and other systems) for this conversion, which may move to seven days a week and three shifts per day.
Using Front Porch Digital as a digital asset management (DAM) system, NASCAR stores the digitized content in a combination of disk arrays and Spectra Logic LTO-4 tape libraries. Data Direct Networks’ high performance storage arrays are used for Apple Final Cut Pro post-production and editing. Data is converted into one LTO-4 tape for digital preservation, as well as one tape for network access.
In addition to high-resolution digitized content, lower resolution proxies are used for editing. NASCAR is now shooting in an all-HD format, with some work using RED ONE cameras at 2K. They plan to move into stereoscopic content in the future. As higher resolutions are used, the storage and bandwidth requirements for archiving and digital preservation will increase significantly.
Lower storage costs will be required to support this growth in archived resolution. In 2006, the Science and Technology Council of the Academy of Motion Picture Arts and Sciences published a report on archiving and accessing motion picture materials. Based on this report, the table “Estimated costs for archiving motion picture materials” (above) summarizes the estimated costs for archiving content on a hard disk array and on tapes in 2006. (The table assumes that disk drive utilization is not 100% and does not include the costs of getting material on the archiving media.)
The storage capacity available on individual disk drives and tapes has increased significantly since 2006, and the other costs in the table mostly scale with the amount of devices or media used for storage.
The figure “Comparison of estimated annual cost” (left) shows projections for the costs of storage for 1PB of content over a 20-year period, assuming a hardware refresh every five years and scaling other costs as appropriate with the decreasing number of hard disk drives (HDDs) or tapes needed. The total cost of saving 1PB of data for 20 years is about $320,000 for disk-based storage and $170,000 for tape storage. Over 50% of the total cost for both disk and tape storage is spent in the first year.
Much content created today starts out in a digital form. Because of the ease of capture and editing with digital content, and the lower costs of acquiring digital content, creators will often create much more content to find the best material for the final product than was the case with analog film or tapes. An example of some archives maintained by content creators:
- 2PB by Weta Digital;
- 6PB by the NBA;
- 850TB by the INA (moving image bank);
- 90TB by BMW (with shadow copy).
Archives can be off-line, as when tape or optical disks are placed on a shelf in a controlled environment, or on a network where they can be accessed more readily. Archived data will not require high-speed access; therefore, if network storage is used for an archive, it is near-line rather than on-line or real-time.
As the cost of network storage goes down and as more data is digitally archived, the percentage of archived data that is in near-line storage will increase. This is due to converted analog and digital data being kept on disk-based storage rather than migrated to tape or optical disk, as well as the increasing value of archived content. On the other hand, the lower cost of off-line storage will be a strong factor preventing more content from being stored on near-line storage.
The figure “Content archiving” (above) shows Coughlin Associates’ projection for the growth of archived storage for off-line and near-line storage. By 2014 we expect 15% of archived content to be near-line storage, up from 10% in 2008. Most near-line storage will be on disk arrays, although some will be on tape and optical disk libraries.
TOM COUGHLIN is president of Coughlin Associates, which provides market and technology analysis, as well as data storage consulting services. Some of the material in this article is from the 2009 Digital Storage for Media and Entertainment report (tomcoughlin.com/techpapers.htm).Coughlin is also the author of Digital Storage in Consumer Electronics: The Essential Guide, and the founder and organizer of the Storage Visions Conference (storagevisions.com) and Creative Storage Conference (creativestorage.org).