Storage requirements for digital content

Posted on September 01, 2005

RssImageAltText

Higher-resolution content is driving the need for huge storage capacities and high-speed bandwidth.

By Thomas Coughlin

Higher-resolution digital content requires larger amounts of storage capacity and higher bandwidth to move these digital assets through the content creation, editing, archiving, and distribution processes. In addition, as entertainment industry files increase in size and number, the complexity of managing, finding, and preserving these files becomes a challenge. This article addresses digital storage trends in production studios, as well as in digital content distribution, and is based on data from the 2005 Entertainment Content Creation and Digital Storage Report, published by Coughlin Associates (www.tomcoughlin.com).

Content creation

Driven by the demand for higher resolution in theatres and homes, feature film resolutions are on an upward roll. In the high-end feature film market 2K resolution is common, and 4K resolution is becoming more common. The table, below, compares some storage metrics for 2K and 4K feature films with a 10-bit file depth. With such growing resolution and storage demands, new solutions will be needed to store and move these assets throughout the studio, into theatres, and into home entertainment distribution.

Digital content acquisition

Most of current feature film production is still done with film. To use modern non-linear editing technologies these films must be converted to a digital form. This conversion is done using devices called film scanners. After the film is finished the digital content must be copied on film again for traditional feature film distribution. The table, below, shows storage requirements for feature film scanning at various resolutions and color depths.

Digital camera developments

An interesting development in content acquisition is the development of digital cameras that use media other than tape. Tape is still dominant and will remain so for several years, but other storage media are increasingly being used in camcorders, such as small-form-factor (SFF) hard disk drives (HDDs). Ikegami, for example, has made cameras containing SFF HDDs for many years.

What follows is the HDV Consortium Specification for 1080i professional digital camcorders:

  • DV or mini-DV cassette media
  • Video signals of 720/60p, 720/30p, 720/50p, 720/25p, 1080/60i, and 1080/50i
  • Pixel raster: 1,280x720 or 1,440x1080
  • Aspect ratio: 16:9
  • MPEG2 video (profile and level: MP@H-14) compression
  • 8-bit quantization
  • Compressed bit-rates of 74.25MHz for 720p line signals or 25MHz for 1080i signals
  • Stereo audio channels (16-bit, 48kHz)

Click here to enlarge image

null

Non-linear editing and special effects

Almost all content creators now use non-linear editing of digitized content, and most special effects are done today with digital techniques. This streamlines the editing process, resulting in faster editing at lower cost.

Non-linear editing is generally done with uncompressed or slightly compressed content since heavy compression increases the overhead of editing and can cause timing problems. The figure on p. 33 is a schematic of a non-linear editing station showing optional connection to shared online storage via a SAN and host bus adapters (HBAs). For a large facility with several editing chairs, shared network storage allows the local disk storage to be kept at about 30 minutes per station. Storage networking has been decreasing in price due to the maturity of Fibre Channel SAN components and growing use of iSCSI SANs and NAS.

Click here to enlarge image

The high-end segment of the non-linear editing market requires expensive components to support bandwidth and latency requirements for 2K and 4K resolution.

Bandwidth requirements for 2K and 4K are shown in the table below. RAM is often used as a buffer in various parts of non-linear editing systems to reduce the impact of system latencies.

Click here to enlarge image

Compositing and special effects are increasingly a staple of films and other digital content. Demand for more-sophisticated results will increase storage requirements (capacity and performance). Special effects are often done with clustered computers connected to storage networks. These are generally based on open computer architectures with proprietary data management software.

Content distribution

Digital cinema offers considerable savings to content creators by significantly reducing distribution costs compared to film prints. Distribution of digital theatre content is through optical disks, hard disk drives, and possibly high-speed download. The potential savings on digital content distribution to theatres is considerable. In addition, digital prints don’t degrade with usage.

Even with the projected savings from digital distribution there has been considerable resistance from theatre owners to install expensive digital projection systems; these systems currently cost about $100,000 for a typical installation, but are becoming less expensive as the technology matures. (Film projectors, in contrast, can last for decades.) Various schemes have been proposed for studios to finance these installations, but these haven’t yet materialized and theatre owners object to potential restrictions on their use of subsidized equipment.

Click here to enlarge image

Broadcast and other forms of digital content distribution use compressed formats to make the best use of limited bandwidth resources.

Distribution methods as well as video on demand (VOD) will require larger amounts of storage as the digital content resolution increases.

As digital distribution increases, cable operators will expand their VOD offerings. This, and the increasing resolutions, will drive demand for digital storage. The table, below, shows VOD storage capacity and bandwidth requirements for standard-definition (SD) and high-definition (HD) digital content. VOD requires that content be ingested as well as played or streamed out. Content is ingested at a rate required to refresh the content in the cache that serves the VOD delivery.

Archiving

Preserving new digital content and converting historical analog content to digital form will be the single largest driver of digital storage capacity. Much of the storage for archiving will be on removable media such as tape and optical disks that can be put on a shelf or in a library until needed. Digital preservation allows content to be available for research and distribution.

Future content businesses will be based on the availability of vast amounts of historical digital content. Digitizing the world’s cultural archives also offers new ways to preserve them for future generations although digital media obsolescence needs to be dealt with as described below. A major obstacle to such digital conversion is the high cost of this effort since it generally requires handling and extensive special manipulation of older film stock and tapes by highly specialized technical professionals.

Many major digital conversion and preservation efforts are underway worldwide. For example, there are very large libraries of material being converted to digital archives, such as the 100,000-hour CNN library.

To take just two examples from major networks and studios, CBS has more than 1,045,000 tapes and more than 150,000,000 feet of film, and Sony/Columbia has more than 35,000 TV programs on 600,000 reels/tapes.

One of the biggest issues for archiving is the obsolescence of the storage media technology. Tapes or optical disks get outdated, and if the digital content that they contain is not transferred to new media it will be difficult to preserve, cannot be easily read, and likely will be lost.

As the size of the digital archive increases it will become more difficult to transfer digital content fast enough to preserve that content. A serious issue in the future will be having sufficient bandwidth available to convert from old media to new media.

Archiving will not be a static or occasional process. Format conversion of large data stores may eventually require almost continuous transfer operations. When the archive load becomes too large, choices will have to be made about which content to transfer and preserve on the new format.

Summary

Between 2004 and 2010 we expect a 900-fold increase in the required digital storage capacity for the digital creation and distribution markets. With the growth in storage demand for high-resolution content and the ease with which digital footage can be acquired, digital storage demands for content acquisition should match those for archiving and preservation by 2010, as shown in the figure, left. We expect that extensive digital conversion projects will occur in the intervening period.

In 2004, we estimate that 60% of the total storage media shipped for all the digital entertainment content segments was on tape, with 40% on optical disks. By 2010, the change in segment demand will also change the mix in digital storage media.

Also, by the same year, tape usage and optical disks should decrease to 40% and 55%, respectively, with hard disk drives comprising a 4% market share.

Tom Coughlin is president of Coughlin Associates (www.tomcoughlin.com), a data storage consulting firm. He is also the organizer of the annual Storage Visions conference as well as co-organizer of the annual Network Storage Conference.


2006 Storage Visions Conference
January 6-7, Las Vegas

Click here to enlarge image

This one-and-a-half day conference focuses on digital storage and its role in the creation and distribution of entertainment content. The event brings together professionals in the digital entertainment and storage industries and provides an environment for networking with professionals who will shape the future of content creation and distribution, consumer electronics, and digital storage. For information on the conference, visit www.storagevisions.com.


Comment and Contribute
(Maximum characters: 1200). You have
characters left.

InfoStor Article Categories:

SAN - Storage Area Network   Disk Arrays
NAS - Network Attached Storage   Storage Blogs
Storage Management   Archived Issues
Backup and Recovery   Data Storage Archives