Apotex leverages SAN-based disk-to-disk backup

A storage area network coupled with disk-based backup slashes backup-and-restore times while minimizing manual tasks.

By Elizabeth Ferrarini

The ability to restore files instantly and eliminate all points of failure in the backup process propelled the CIO of a large Canadian pharmaceutical firm to abandon tape media for network backup. Michael Davidson, CIO of Apotex, a generic drug manufacturer and biotechnology firm based in Toronto, proposed a disk-to-disk backup strategy for distributed servers at the parent company and its two affiliates. Two identical storage area networks (SANs), scalable to 8TB each, currently back up Windows NT/2000 SAP servers as well as Microsoft Office file servers across the three organizations' campuses located around Toronto. Apotex has more than 4,000 employees and 2 million sq. ft. of facilities around Toronto.

A few years ago, Apotex's backup environment was a mess. "We did backups upon backups at each building," says Davidson. About 500GB of data got backed up on 70 distributed servers running a variety of operating systems, including VMS, NetWare, Windows NT, and Unix. Each operating system had its own backup software and hardware, comprising about 15 DLT and DAT tape devices. The media library consisted of thousands of tapes—some of which were 10 years old—which had to be maintained throughout various retention cycles. "At one point, we were spending an incredible amount on media—$250,000 a year—just to have all of these tapes sit in a third-party's off-site facility," Davidson adds.

Naturally, he wanted to eliminate the costs of buying media, storing it off-site, and licensing backup software. Cutting the restore time of a file from two days to 10 minutes would provide significant cost savings and was one of the biggest reasons to move away from tape. "Every day we faced steep intangible costs as well as the uncertainty of being able to restore files from tapes going back 10 years," Davidson explains. "And we had the frustration of dealing with tapes for different operating systems and different pieces of hardware."

As a result of installing a new SAP ERP system in 2000, Apotex consolidated to two server platforms—Windows NT/2000 and Solaris—rather than four. The storage capacity more than doubled to 1.6TB of data to back up. "However, we still had 60 distributed servers and no consolidated backup solution for Apotex and its affiliate companies," says Davidson. "We asked ourselves: 'What backup technologies could we use to conserve storage space, reduce the backup window despite capacity growth, cut restore times, and minimize media and off-site storage costs?' "

Online Disk-to-disk Backup
Apotex's IT department first tested disk-to-disk backup of the Windows NT servers via the Internet to EVault (formerly VytalNet), a storage service provider in Mississauga, outside of Toronto. However, says Davidson, "it was too expensive to push terabytes of data to a service provider each month. It made more sense to build our own system and become our own disk-to-disk backup provider."

Last year, the IT department went live with the backup of its Windows NT/2000 servers in the first phase of its disk-to-disk backup system. The second phase will include the Solaris servers.

The data center in Toronto has a Compaq ProLiant 530 server connected via Brocade Fibre Channel switches to a Compaq EMA 1200 StorageWorks RAID subsystem. This SAN can scale to about 8TB. Currently, the SAN has about 3TB of used space. An identical secondary SAN resides in an Apotex affiliate's data center about 30 miles away. This SAN acts as an alternate system in the event of failures in the primary SAN. A second affiliate, which doesn't have a SAN, is also involved in the backup system. A 100Mb fiber-optic data link connects the three sites.

Disk-to-disk backup software is installed on each of the SAN servers. Backup agent software is also installed on each of the servers to be backed up, as well as on the workstations used for initiating backup-and-restore functions and receiving messages about the backup tasks.

Backups run in parallel at scheduled times during the night. All servers with the backup software agent back up to the primary SAN in the data center and, at the same time, to the secondary SAN at the affiliate company. "This procedure enables us to have two SANs with identical copies of the backup data online at all times," Davidson explains. "We can easily retrieve our backed-up data, and we also have a copy of the data stored off-site for disaster-recovery purposes."

The backup software provides the key ingredient to Apotex's disk-to-disk SAN backup system. Apotex licensed the Windows-based software EVault developed for its online backup service. EVault, which is now located in Walnut Creek, CA, had been offering disk-to-disk backup services for about six years.

EVault's technology shrinks the backup window significantly by copying just the changes, not the entire file, since the last backup for that server. From the initial backup on all servers onto the SANs, the software creates a map in an index file. Each time a backup occurs, the software looks for files that have block changes and then locates those changed blocks, compresses and encrypts them, and sends them over the wire. "A 50GB file now takes about 20 minutes to back up rather than several hours using tape," says Davidson.

Restores have gone from being an onerous task to being as simple as cut and paste. "Within a matter of minutes, as opposed to days, we turn around requests for restores," Davidson reports.

The new disk-to-disk backup system has eliminated a lot of repetitive tasks that IT administrators used to perform. These individuals no longer have to do manual tape intervention tasks such as mounting tapes, sending them off-site, and retrieving them.

Besides disk-to-disk backup, the SANs provide other types of redundancy to improve data protection. "We can configure and partition the SAN to contain a replication of the data stored on any of our business-critical servers," Davidson says. "We can perform the replication in real time or schedule it as an overnight process. We can also partition the SANs as an archival vault for major applications such as SAP."

Elizabeth Ferrarini is a freelance writer in Boston. She can be reached at iswive@aol.com.

This article was originally published on April 01, 2002