The media and entertainment industries weren’t always big users of storage technology. They were accustomed to shooting film on celluloid or video tape. Old film and tape was simply stored on shelves or in vaults. But then came digitization. Suddenly data storage was a big deal. The solution in the industry has been to use disk but that approach is now proving too expensive.
Storage industry analyst Tom Coughlin of Coughlin Associates explained the challenge. Frame rates for movie content are increasing from the historical 24 frames per second (fps) to 48 and 60 fps, and may eventually go as high as 300 fps. And as resolutions get higher, file sizes soar. Per hour of film standard definition (SD) is 112 GB, HD is 537 GB, Ultra HD is 6,880 GB, 6k is coming onto the market, 8K is on the near horizon, and some are contemplating 16k. When 8k appears, each hour of film will amount to 86,000 GB of capacity.
In other words, 8K film would use more than 100X the capacity of HD, said Coughlin. And then there is free viewpoint imaging which uses many cameras simultaneously to get the viewpoint of the film to change.
“As video resolution and frame rates increase, camera image complexity rises and stereoscopic projects multiply, the storage capacity and bandwidth performance requirements become staggering,” said Coughlin. “How long will it be till we see Exabyte-scale video?”
16k film would eventually call for data rates of 115 Gb/s and would generate about 414 TB per hour of content. But if four cameras are used to create data for a free viewpoint presentation, the raw data could add up to 1.66 PB per hour of content.
This frightening quantity of data is driving movie companies into the cloud. According to a survey by Coughlin Associates, 30.2% of media and entertainment companies said they used cloud storage for post-production work in 2015 versus 25.6% in 2014 and 15.15% in 2012. Further, 32.9% said they had 1 TB or more of storage in the cloud in 2015 versus 23% in 2013. And more than half capture six hours of content for every hour of movie. For some it goes as high as 50 hours.
The sheer amount of content makes it vital that movie houses clearly define what content should be kept to hand for rapid editing. How this works out is that brand new content for films and shows is heavily accessed until the work is completed. Once completed and distributed, that content suddenly goes dormant and is rarely accessed thereafter.
Coughlin said 95% of all content, therefore, resides in massive archive. However, in terms of dollars, the archive is only allocated 45% of the available storage funds. Although this material is archived, it can be called upon on occasions such as anniversaries, documentaries, breaking news and for marketing purposes.
“This makes the concept of an Active Archive appealing to the media and entertainment industry,” said Coughlin.
To achieve this, a variety of media are employed. Optical disk, tape and Hard Disk Drives (HDDs) are used together, depending on the required response time and touch rate. The mechanics are simple: As response time requirements move closer to instant and the volume of access grows, costs soars as the fastest storage is called for; as response time lowers and touch rate declines, slower and cheaper storage comes into play, based on touch rate and pre-set policy.
HDD will clearly play a role in such an arrangement, supported by cheaper optical disk for faster access/low capacity and tape for higher capacity/slower access. Among media and entertainment today, Coughlin said tape was the highest media in use for archiving at 40% (with a growth rate of 59% expected this year), followed by HDDs at 28%, local storage at 16% and optical at 6%.
In terms of capacity market share, HDD owns 48.9% while tape has 40.4%, optical 10.15% and flash 0.6% for the industry as a whole. Much of this storage is migrating from the internal data center to the cloud. “Cloud storage for media and entertainment will exceed $2.1 billion by 2020,” said Coughlin. “This will be mostly for archiving and will primarily be comprised of tape and disk.”
Aaron Brenner, Executive Director at sports TV production company Showrunner, was a big fan of the storage area network (SAN) and was happy to gobble up disks for content storage. As the SAN filled up and content had to be archived, hard drives were taken out of their slots, labeled, logged and stored on shelves.
“We need to save sports content forever as it can be monetized by using clips again and again,” said Brenner.
Back in 2008, only 100 GB of standard definition footage (720 x 480) comprised the entire library of LA Kings footage that Brenner worked on. By 2015, every game produced 65 GB and the library expanded to half a PB. For example, a 13 episode Stanley Cup video series took up 100 TB of raw media.
“Storing disk drives on shelves gave us no redundancy, took too much time and became too expensive,” said Brenner. “But at the same time, the industry had been going tapeless for years by getting rid of video tape so we were averse to tape storage.”
The workflow at Showrunner now comprises camerawork storing directly onto flash. Those flash drives are downloaded onto disk and then fed into the SAN until production and post production work is complete. From there, the material is put onto a Dternity hybrid storage system made up of disk and tape. This means the company can access archived clips much faster with no need to rifle through piles of hard drives. Tapes are also vaulted offsite for DR purposes.
“When we are ready to archive, content gets dragged and dropped onto the Dternity, we verify it is there then delete from the SAN,” said Brenner. “Tape allowed us to focus on being creative and not worry about archiving or having enough disks to store our data.”
A similar workflow is used at color grading company EFilm. Data captured from cameras goes onto the SAN, and once the color work is done it then goes onto LTO tape for archiving. The company makes three copies of every LTO cartridge: One stays on location, one is sent to the office and one is sent to a secure location for DR purposes. This is done due to the extremely high cost of a day’s shoot.
“We have about 3 TB of disk onsite for editing and color correcting of film,” said Josh Haynie, Senior Vice President of Operations at EFilm. “But we might use a couple of thousand LTO cartridges for every big movie project.”