Techniques for Storing and Delivering Video
RAID-based video storage and delivery requires compression and in some cases a Fibre Channel network.
By Martin Bock
If it`s true that a picture is worth a thousand words, then a video may be worth ten thousand. Unfortunately, in data storage terms, a file of that size can pose major challenges. A single word may fit into eight bytes, but a five-minute, heavily compressed video segment may require 52MB. This order of magnitude difference is a real challenge for storage solution vendors, requiring new methods for storing, retrieving, and distributing video content. The first step is video compression.
In the above example, a five-minute video can be squeezed into 52MB of disk space, but only if it is compressed. A standard uncompressed digital video format consumes more than 200 times that, or roughly 10GB of disk space. Thus, compression techniques are vital for most video storage applications.
Several video compression techniques are available to assist in the storage dilemma. The major differentiation among techniques is the compression ratio (see table). The storage capacity requirements for MJPEG and MPEG2 represent averages. These compression techniques support variable bit rates to further reduce capacity requirements and transfer rates.
Data rate is another challenge for storage vendors. Retrieving a word of data is typically an I/O-intensive activity: the more I/Os that are processed per second, the more individual words that can be retrieved. Video, however, is a bandwidth-intensive (or transfer-rate-intensive) activity. When a storage system retrieves a video, few I/Os are required, but data must be continuously transferred. In the case of a five-minute video clip, accessing data may require one I/O command, but the transfer of data continues for five minutes, typically dominating the data path to the user for that period. The table lists data rates for single videos. Data rates increase rapidly when video is served to multiple users simultaneously.
Why So Many Formats?
Video compression is not a perfect technology, however. None of the compression methods listed in the table are "lossless" compression techniques, which means video applications will lose data. Aside from medical imaging, most video applications can operate with some data loss--an acceptable compromise to reduce speed and capacity requirements by up to 200 times.
But some applications are less forgiving than others. Data loss affects resolution and clarity. Some applications allow for larger degrees of resolution loss than others, while some do not tolerate resolution loss at all.
Applications requiring higher picture quality benefit from YUV or MJPEG compression because of the lower compression ratio. (The lower the compression ratio, the higher the picture quality.) Other video applications, which depend on picture quality, can use MPEG2 or MPEG1 compression.
Post-production servers and in-flight video servers are at opposite ends of the picture-quality spectrum. Post-production servers used in the feature-film industry require maximum video picture quality. Video is usually digitally encoded in lossless (uncompressed) format so that the pictures can be blown up and displayed at big-screen theaters.
This work can be created in very high-resolution format, and at 24 or 30 pictures per second. High-definition requires a data rate of up to 248Mbps.
In-flight video servers, in contrast, do not have such high picture-quality standards, so video can be heavily compressed. Since the picture screen is a small six-inch monitor, a few missing frames of data, which degrade resolution, are not noticeable. For these applications, MPEG1 compression is sufficient.
Other applications for video servers fall somewhere in-between the above examples. Hotel video servers and training videos for medical applications, for example, typically use MPEG2 techniques. Kiosk video displays, music videos, and some corporate training video servers often use MPEG1.
RAID for Video
Compression makes the handling of video data manageable, but not necessarily easy. For server applications--not for typical reasons, though.
In video applications, the value of RAID is its ability to ensure data access (i.e., system up-time) and also to guarantee the stream of data is sufficient to play the video continuously and meet the higher data rates required by multiple simultaneous video transmissions.
According to industry reports, RAID-3 best handles large files such as video, while RAID-5, which has traditionally been thought of as a transaction processing storage device, is ideal for small records. In video server environments, however, both RAID-3 and RAID-5 are effective storage configurations--if they are matched to the application.
The higher the compression ratio, the smaller the video frame. For example, an uncompressed single frame of data is typically 1MB. When the frame is compressed using MJPEG (6:1 compression ratio), it is reduced to 166KB. That same 1MB frame reduces to just 5KB of data with MPEG1 compression.
So, uncompressed and lightly compressed techniques require RAID-3 because large blocks of data transfer most efficiently over RAID-3`s striped disk design. With MPEG1 compression, a single frame is actually a small block of data. As a result, depending on the degree of randomness of frame retrieval by the video application, RAID-5 could be a better candidate because it handles small blocks more efficiently than RAID-3.
Another point to consider is the RAID array`s ability to maintain speed. A RAID subsystem that guarantees a specific data rate in the event of an error or should a disk fail benefits highly compressed and lightly compressed video server applications.
Video data must continuously transfer at a prescribed speed or the data stream ceases to be full-rate video--frames are dropped and the smooth video image transforms into a jerky series of pictures.
A traditional server of text and pictures, on the other hand, can tolerate performance degradation; data is simply delivered more slowly to users--a minor inconvenience in most scenarios.
Further, if transfer rates are predictable, video server designer use less hardware. However, if transfer rates degrade, designers must overbuild the system storage to account for the worst-case transfer rate of the problematic RAID device.
Regardless of the compression rate, video presents an enormous bandwidth problem when multiple streams are requested simultaneously by clients on the server. This requires a storage solution that can accommodate high-bandwidth continuous transfers (see figures).
These figures illustrate two video server configurations: one for lightly compressed data, the other for highly compressed video. One application uses a combination of Ethernet and coax cable in a desktop video server configuration; the other uses Fibre Channel in a high-quality video-editing system.
The desktop server example illustrates careful handling of data to ensure consistent delivery. Three separate paths are used for content load, video requests, and the actual video delivery path. This architecture has several advantages.
- Minimal downtime. The server can access and compress and store new video content at the same time.
- Lower costs. Since the host processor is not in the high data-bandwidth path, it can be a low-cost PC. The host uses existing Ethernet communications for video selection through menus. Video content is decompressed prior to delivery to users and is delivered over standard low-cost coax cable. Because the video decompression activities are centralized, each user does not have to handle the decompression, which means PCs or standard televisions can be used at the client`s location.
For higher-quality video applications, those requiring video editing and file sharing, a faster network is required. These applications could benefit from Fibre Channel connectivity to provide a very high-speed network.
As in the desktop application, the video content typically starts in analog form, in this case film or video tape. The film or tape is digitized and sent to the server. Now that the content is digital, it can be presented to users in an editable form. In this scenario, each can view and manipulate the data to create special effects or to insert 3D animation and graphics.
The Fibre Channel network can be constructed using hubs (or loop configurations for smaller 100MBps networks) or fabric or Fibre Channel switch technology to create multiple simultaneous 100MBps video paths. Providing shared video in a video-editing application increases workflow and reduces processing time.
Video sharing is the next evolution in the distribution of information to the desktop. The challenge of moving capacity- and throughput-intensive video to viewers is an order of magnitude more difficult than providing text and still pictures. RAID storage solutions are key, as is Fibre Channel, especially in high-quality video applications.
In hotel video server and corporate training applications, networks can be configured with standard Ethernet and coax cables.
In applications requiring high-quality video, as well as video editing and sharing, Fibre Channel provides a high-speed network.
Martin Bock is vice president of marketing at Storage Concepts Inc. in Irvine, CA.