Custom Revolutions requires custom storage solutions
The customdisc.com Web site needed a roll-your-own combination of RAID, optical and tape libraries, and special software.
By Zachary Shess
Due to the unique properties of their data, e-commerce firms frequently do not use off-the-shelf devices or software for their storage needs. For Custom Revolutions, in Stamford, CT, "customized" best describes its business and its storage configuration.
Custom Revolutions operates customdisc.com, a Web site where visitors can go to make and then buy customized music CDs. The site, which has a library of more than 185,000 songs spanning every music genre, receives about a million weekly page visits and generates thousands of CD shipments every day.
Downtime for any young e-commerce company can equate to financial disaster. If customers can`t access the site`s contents, money cannot be made.
To store, retrieve, and protect customdisc.com`s 185,000 songs and other critical data (more than 8TB), Custom Revolutions and its IT consultant, Fairfax, VA-based American Management Systems (AMS), identified the company`s top storage requirements as fast data access and scalability.
"The biggest challenge is that every single song needs to be online or nearline for customers to always have access to make CDs," says Custom Revolutions president Nick Darveau-Garneau.
"There`s no 80/20 rule that applies here. There aren`t that many songs selected all the time versus songs that aren`t selected as often," says Darveau-Garneau.
After initially serving music requests by manually pulling CDs out of racks and placing them into jukeboxes, Custom Revolutions wanted a faster robotic solution to move CDs from jukeboxes to CD-ROM readers.
Unable to find an off-the-shelf solution, Custom Revolutions and AMS started down a path toward building a customized, highly scalable storage configuration.
Over the course of more than two years, the two companies assembled a hierarchical storage management (HSM) configuration. The three-tier system includes RAID arrays, CD jukeboxes, and DLT tape libraries, which are linked over a 1Gbps IP network. The amount of storage in each system varies depending on selection frequency by customers.
However, Compaq RAID arrays hold about 1TB of the most frequently selected music, while DLTtape libraries hold the least-requested songs.
Most of the music is housed in several Plextor 4X MegaPleX CD-ROM jukeboxes networked to Plextor`s customized PleXServers, which are optimized for high-performance audio extraction. Custom Revolutions found the jukeboxes to be easily scalable and less expensive than hard drives. In fact, Darveau-Garneau estimates they have saved $500,000 by using jukeboxes.
Tempering the promise of HSM over the last few years has been a lack of off-the-shelf software to manage complex and divergent tiers of storage. Custom Revolutions and AMS recognized this problem.
To meet the business objectives and the time-sensitive nature of the customdisc.com`s on-line music business, they opted to develop customized software. Pressing software issues included data management and a fast, reliable file system for data extraction.
AMS spent two years and one million lines of code to develop automated data management software. Each week, the software sweeps all the tracks, determines which songs are most popular, and stores them accordingly. Popular songs are then automatically retrieved and cached onto RAID arrays.
Custom Revolutions and AMS approached Plextor to help them create a file system for multiple jukeboxes. Somewhat serendipitously, Plextor at the time was working with Germany-based iXOS to develop such a file system.
Later named Jukeman, the suite provides quick song delivery through automated data extraction, while streaming audio at drive speed through a file system spanning multiple jukeboxes, according to Vadim Brenner, a product manager at Plextor.
"Each CD shows up as a subdirectory under a single drive letter. So you can have thousands of CDs under one drive letter," Brenner explains. "It`s part of the native NT file system. Therefore, any abilities that NT has are inherent to our solution."
Jukeman automatically streams tracks off drives in multiple jukeboxes and converts the audio .wav or .aiif files onto RAID devices in real-time. Selected music is then sent over the network to specialized Rimage Corp. CD-R jukeboxes, which then "burn" the music onto a CD. A CD is produced within 20 minutes of being selected and then shipped to the customer.
Network throughput and data retrieval performance are excellent, says Darveau-Garneau, crediting the storage configuration and the huge music library as two of the company`s primary competitive advantages.
Acknowledging that not too many IT administrators are assembling or managing storage environments that are similar to a custom CD web site, Darveau-Garneau says common issues merit consideration and can transcend the type of business and storage environment.
For example, he says the company`s success in part came from paying close attention to storage usage patterns and tailoring backup and retrieval activities to optimize those patterns.
"Getting this up and running is not the hardest part. It`s managing all this. You really need to look at the data and usage patterns," he says.
Because of the size and complexity of assembling a multi-terabyte storage environment, Darveau-Garneau is a firm believer in surrounding yourself with good help; in this case, securing the right consultants and vendor partners. "Whenever you give a key part of your system away to someone else as part of your overall solution, it`s going to be much more than a handshake. We were fortunate to have great partners."
Custom Revolutions` storage configuration
Due to competitive concerns, Custom Revolutions officials declined to name the specific number of devices used to store music for customdisc.com.
The primary components used to store and retrieve tracks and produce customized CDs include:
- Compaq RAID arrays
- Plextor MegaPleX 4X CD-ROM libraries
- Plextor PleXServers
- JukeMan file management software
- Rimage CD-R jukeboxes
- DLT tape libraries