Digital video is growing in importance behind many types of different use cases today, yet nowhere is it more important than when it is captured, edited, and produced into a revenue-generating product for a video business. Within such organizations, video becomes the very embodiment of intellectual property – every piece of content may be viewed as a key asset that can be time and again reused or reprocessed and consequently be the source of new revenue in the form of additional final products, broadcasts, distributions, partnerships, or more. Such businesses have long been immersed in the practice of reusing digital video – typically in a cycle where a single piece of media may be recalled, redistributed, re-edited, or otherwise brought freshly back to the forefront of the business time and again.
But turning to the reuse of digital video hasn’t been a process that organizations could undertake overnight. While on the surface it seems possible to rush forward and acquire sophisticated digital capture and editing capabilities, unwary users will quickly run into long-term challenges. Those challenges are most often rooted in storage. Specifically, the intersection of digitized video, increasing resolutions, more frequent reuse, and sophisticated post processing capabilities have caused video sizes to burst at the seams – often pushing individual projects into terabytes of data behind the guise of one digital video.
Moreover, organizations today are creating multiple resolutions of video for use in different situations – a single video asset may be transcoded at web resolutions, standard definition resolution, multiple high definition resolutions, and even 3D. Pure un-encoded 1080p HD video at 24bpp and 60fps easily exceeds 350MBps of bandwidth. While encoded and compressed files may drop to 40MBps to 50MBps, single two-hour long files can easily become 100GB+ monsters, and 3D video may require several times this.
In addition, video made up of randomized data receives no benefit from data deduplication technologies. While it may be possible to build a system sufficient for enormous video projects, or several video projects, reusing video over time can rapidly become impossible without careful forethought.
With the challenges of both creating and then storing enormous amounts of video at the forefront, video organizations have settled on a split storage infrastructure, and rightfully so. In one tier, high-speed production capture and editing takes place. In this tier, the organization invests enormous amounts of money on performance.
But long-term storage of video takes place in another tier – the archive. While this tier is cost-optimized for storing enormous amounts of data, the video professionals are well aware that this efficient repository is responsible for protecting the crown jewels of the business.
But the coexistence of these two worlds has often left a gap. Archive systems, with a focus on deep data integrity and scalability, are frequently plagued by architectures that were designed with the expectation that rapid access would never be a requirement. Storing and recalling terabyte-size projects that may span many pieces of media may create complex processes and waste hours of idle time while staff waits for the retrieval of a project. Yet the speed and performance of primary storage systems are extremely costly, and scaling those systems to potentially support petabytes of mostly idle archived video is unachievable or unfeasible for most organizations.
Entertainment/media organizations are stuck between a rock and a hard place. Either over invest in expensive storage that is misused for archiving, or spend enormous amounts of time waiting for slow archive systems to regurgitate video assets as they are needed.
Clearly, storing and reusing digital video over a long period of time would be aided by a mix of some of the best characteristics of both production (primary) and archival (secondary) storage.
The Active Video Archive
Fortunately, the storage industry is innovating in response to the demands of media-intensive businesses. The focus for a select set of innovators rests on delivering a better video archive – one that can blend the low cost, high data integrity, and long-term retention of the traditional archive with the higher performance of conventional disk-based storage. Such solutions can recall data to production storage with production-like efficiency, while maintaining all of the integrity, scalability, and protection a long- term archive requires.
Many solutions augment deep tape archives with disk, or better yet, deliver cost-effective disk-based solutions that can still seamlessly migrate their stored data onto tape when it becomes inactive. Such solutions are creating what the Taneja Group refers to as an “active video archive.” For video professionals, the benefits can be tremendous, as they turn the archive repository into a more active storage layer that can immediately serve up archival content for reuse while giving up none of the high integrity and low cost benefits of traditional archives.
Discussions with end users have identified the best-in-breed capabilities for the next-generation active video archive:
? Resilient – An active video archive must be able to guarantee the integrity of a video asset across what may be decades of reuse. Few digital storage systems are designed to store data for the period of time over which video may be reused. Beyond high availability RAID and redundancy technology, active video archival systems today go even further with deep check-summing and file segment mirroring technologies that can deliver integrity far greater than the best of traditional disk or tape subsystems. A disk-based archive can interactively leverage these advanced capabilities to constantly manage the integrity of the system – automatically policing against corruption and recopying data or otherwise correcting for data problems as they occur.
? Scalable – The active video archive must be able to scale in both capacity and performance to handle the expanding size and seemingly endless creation of digital assets. As experience with digital content storage dictates, it is nearly impossible to plan for tomorrow’s storage demands. Video organizations should look for a storage solution that can grow beyond their most aggressive expectations. Moreover, businesses should be attentive to performance as an important dimension of storage growth – as assets increase, the number of users and systems recalling and reusing data will only grow as well. Whereas many traditional archive solutions could easily become bottlenecked when trying to access multiple pieces of media serially, newer solutions can scale and parallelize performance by adding additional nodes to a system. Moreover, some vendors can intermix different generations of hardware and software, allowing users to add capacity and performance without upgrading the entire archive. Such scalability over the long term can create a “scale forever” single archive free from media or hardware upgrades. Some solutions can even create freedom from vendor-specific hardware dependencies through software-only offerings that can scale seamlessly across vendor-agnostic industry standard servers and hardware.
? Open and accessible – The active video archive must be easily accessed across multiple projects and systems, with speed. Many traditional archives have not only slowed down the performance of storage and retrieval with serialized access to media, but such systems have serialized access as well, by requiring specialized software for access or requiring organizations to route all access through master servers that control and manage the only interface to the archive. Modern Digital Media Asset Management (DMAM) systems have opened up access to the archive, turning archive management into a task that takes place outside the storage repository itself. Consequently, an archive that can be accessed over open protocols such as NFS or FTP can allow many clients to simultaneously access an archive and reap enormous benefits in speed and efficiency.
? Affordable – With immense amounts of long-term storage at the forefront of the decision making process, the affordability of storage in a digital video archive is seldom overlooked. Today, the same disk-based systems that increase responsiveness are cost competitive with the best solutions available for archive. Moreover, cost effectiveness at scale is critical. Active video archives maintain their cost effectiveness, as they do not require additional entire libraries or new sets of drives to read media. Over time, disk-based solutions only maintain their cost effectiveness by realizing the benefits of constantly dropping hardware prices, and ever increasing disk capacities. By simplifying and harnessing the best capabilities of multiple types of media – tape and disk – a single well-integrated active digital archive’s value proposition and flexibility can only increase.
In our view, this list of capabilities exemplifies the best-of-breed solutions coming to market. These innovations around digital video archival storage position such solutions to significantly enhance how organizations work with their video assets. Not only can the best of these solutions still offer the same scalability and integrity that have always been required for long-term archive, but they can do it with higher speed disk-based architectures that will convert the dusty inaccessible archive into an interactive tier of storage that is instantaneously responsive to demands for content.
Moreover, the best solutions often blend disk and tape together behind a single repository that masks the complexity of migrating data from a higher performance disk tier to a longer term dormant tape tier, while keeping power, performance, and capacity optimally balanced.
Benefits
For organizations dealing with digital content, an active video archive can have a significant impact on the cost of doing business, in several ways:
1. By reducing the idle time that key staff now spend waiting for slow digital video access.
2. By making video content more accessible, and thereby more likely to be reused as an asset in other projects, or for additional revenue generation.
3. By reducing administrative overhead as the active video archive simplifies complex archival and retrieval processes.
4. By setting the business free from periodic hardware and media replacement that may be costly and disruptive to the organization.
Solutions
In the leadership positions when it comes to active video archive built on scalable disk and designed as extensible video repositories, we see vendors such as Active Circle, Avid, Front Porch Digital and Omneon standing out where media businesses are leveraging storage technologies to solve their challenging video archive management problems. In addition, a number of scale-out NAS solutions — such as IBRIX from HP, Scale-Out NAS from IBM, and ExaNet from Dell — may in the future be able to fit the bill, although these solutions are not focused specifically on video.
Moreover, as the cloud continues to evolve, it is possible to leverage solutions such as EMC’s Atmos, Iron Mountain’s VFS and Nirvanix’s CloudNAS, to name a few examples, as a tier within a larger video archive. However, for many of these cloud-based solutions, special efforts must be taken to leverage them as a fully workflow-integrated archive. Such cloud solutions often must be carefully engineered to interface with the lifecycle of video management processes.
In our view, some of the younger vendors in this space – such as Active Circle – as well as potential future offerings from scale-out NAS vendors – may offer truly differentiated values in cost of capacity, protection, and extensibility for next-generation active video archives.
Recognize that some vendors are turning to disk as a caching layer to offset the administrative and lack-of-responsiveness issues inherent in tape, while other vendors have an end-to-end solution designed as a truly active video archive. Depending on which approach a vendor has taken, a solution may be more or less scalable, and may be more or less capable of delivering the key benefits.
