Utility ensures 24x7 operations with RAIL backup
Multiple backup tape copies and a redundant array of independent libraries (RAIL) provide continuous operation and data protection.
Ameren--formerly Union Electric--is one of the country`s largest public energy utilities, serving more than one million customers in Missouri, Illinois, and Iowa. As with any utility company, Ameren is critically concerned with keeping operations running 24x7, placing an extremely high priority on systems and data availability. To meet that objective, Ameren`s
corporate policy requires duplicate copies of all backup tapes so that a copy is kept safely off-site in case of a catastrophic data loss.
Ameren`s computer installation at the company`s St. Louis headquarters encompasses 1,500 nodes with 120 heterogeneous file servers running Windows NT, NetWare, Solaris, and HP-UX. All departmental data, along with custom applications developed for time reporting, customer service systems, and outage management is backed up nightly to two centralized tape servers.
Ameren uses two Hewlett-Packard LXe Pro servers operating as backup servers, with each server backing up 60 file servers. The installation generates a nightly incremental backup load of up to 100GB and a weekly full-system backup of 400GB to 500GB. Each centralized HP tape server is connected to two Storage Technology 9714 tape libraries, equipped with 4 or 6 DLT7000 tape drives and 100 tapes. For backup software, Ameren uses Legato NetWorker.
Ameren`s IT staff tried to satisfy the corporate dictum of duplicate backup tapes with NetWorker`s tape cloning feature, which provides off-line tape-to-tape copy functions after the backup process is completed.
The tape-cloning feature enabled Ameren to create a set of off-site backup tapes, but not without a hitch, according to Vance Bufalo, senior engineer in networking engineering at Ameren.
"My servers are doing so much work every night, and the off-site tape has to be ready for pickup by 11 a.m. I didn`t have time to do the backup at night and do the tape-to-tape copy in the morning before the vendor showed up to pick up the tapes," Bufalo explains.
"The other problem I ran into is that the two tape systems support approximately 1,500 users in our main office complex, and we do recovers on a daily basis. If the tape drives are tied up doing tape-to-tape copies during the morning, they are not available for tape recovers. I needed a solution that provided real-time tape mirroring. As one tape is being written, a second tape is being created."
The need to do real-time tape copying led Ameren to Ultera Systems (Laguna Hills, CA) for a real-time tape mirroring solution centered on Ultera`s ShadowMaster tape controller technology. A mirrored tape architecture essentially creates a RAID-1 environment by simultaneously writing data to a mirrored set of tape drives. Automatically creating a duplicate set of tapes for off-site storage in the time it takes to do the original backup solved Ameren`s backup window problem.
Ultera`s tape mirroring controller technology was also appealing. Because the controller is software independent, Ameren could continue to use Legato without any changes. Incidentally, the controllers work with leading backup software applications for Unix, NetWare, and Windows NT.
For the Ameren installation, the tape configuration goes beyond simple drive mirroring and includes RAIL (Redundant Array of Independent Libraries). The RAIL configuration mirrors a pair of STK 9714 libraries. One library is dedicated to storing on-site tapes and the other is used exclusively for tapes created for off-site storage. Each pair of mirrored drives is driven by a dedicated ShadowMaster controller. Another ShadowMaster mirroring controller manages the robotic media changers in each library.
This approach not only enables real-time tape mirroring, but also makes media management painless. "We have a 30-day tape rotation because that`s our data retention policy, and tapes never leave the on-site library," says Bufalo.
Ameren`s RAIL solution is designed so that it is impossible to perform backup operations without creating the required off-site set of tapes. Read operations read data from tapes in the primary library drive, but if a write command is issued, the ShadowMaster controller automatically writes data to the two mirrored drives. If the second drive is not ready, an error message is generated and the backup process is aborted.
"That gives us real-time mirroring, which meets our requirements. Off-site copies are ready by 11 a.m., and we can still perform data recovers from the on-site tape library for the last 30 days without any tape handling by the operators," says Bufalo.
The library mirroring solution worked so well for Ameren that it recently implemented a larger configuration, deploying an identical mirrored-library setup using Ultera controllers with an HP9000 K-Class Enterprise Server running HP-UX with two STK 9710 libraries. Each library accommodates 512 tape slots and 10 DLT7000 drives.
Still, Ameren has even bigger plans for its tape libraries. The next step will address the ever-expanding data requirements, not just backups of the corporation`s 5,000 nodes. The current development plan is to incorporate Ultera mirroring controllers with large-scale STK Powderhorn libraries.
Real-time tape- mirroring controllers simultaneously write data to two tape libraries, creating a duplicate set of tapes.
Dan Murphy is a freelance writer specializing in storage, in Redondo Beach, CA.