Utility provider solves backup nightmare

Backup solution is based on an unlikely tape format-DTF-2-but despite upfront costs, the utility expects to save $300,000 over the next three years.


In the fall of 2000, the Public Service Company of New Mexico (PNM) was trapped in a classic enterprise backup nightmare. Following years of steady growth, New Mexico's leading supplier of gas and electricity had 86 Windows NT servers on its network, each with its own dedicated DLT drive. Every night, the staff spread out over the enterprise with shopping carts, replacing tapes and backing up servers one by one. Sometimes they finished by morning, but often they didn't. Each backup had to be checked manually for errors, on a server-by-server basis-another drain on PNM's network staff.

"At any given time, we had no more than four days' worth of backup on-site," says Don Atwood, PNM's senior information center specialist. "Everything else was off-site and would take at least 24 hours to retrieve and restore from the recovery facility."

Do the math

PNM arrived at this impasse in the classic way: by implementing backup on a server-by-server basis, until things got out of control. "We had a centralized backup plan on the drawing board for more than a year, but it was a half-million-dollar project, and we kept canceling it for budgetary reasons," Atwood explains. "It was cheaper to add a $6,000 tape drive to each new server, so that's what we kept doing." Then a spike in PNM's projected network server count, as well as rapid data growth, forced a change of strategy. "Once we determined that we'd be adding 100 new servers in the next four years, suddenly the half-million-dollar solution looked cheap."

Atwood put out RFPs, and fifteen integrators, resellers, and OEMs responded, including Compaq and IBM. "We knew the big vendors would have backup solutions, but we wanted companies that specialized in backup, installed backup for a living, and spent their days and nights taking backup support calls. That narrowed the field in a hurry."

One of the bidders who made the cut was Intelligent Solutions, a Medford, MA-based systems integrator that Atwood found while searching the Web for backup equipment information. Intelligent Solu tions' proposal was built around Sony's PetaSite Scalable Mass Storage System and DTF-2 (digital tape format v.2) technology. The PetaSite library is common in the broadcast industry but is a rarity in commercial enterprise networks.

"The DTF library's throughput [605GB per hour, sustained] was between four and five times faster than the fastest DLT option," says Atwood, "and the library's native capacity [96.6TB] was almost double anything else out there." Still, Atwood was hesitant to be an early adopter, but not for long. "If you're going to spend half a million dollars on a backup solution, you should get the best technology. The DLT solutions that the other companies proposed had throughput and scalability limitations compared to DTF."

Ease of use was also a key criterion. "Our goal was to enable backup of the entire enterprise by one person, from one GUI console, to one set of tapes," says Kevin Metcalfe, president of Intelligent Solutions. "In the morning, all the manager has to do is check his e-mail to see which backups were successful and which weren't."

Once Atwood chose the backup solution, integrating it with PNM's network began. According to Matt Rock, Intelligent Solutions' director of engineering services, "Implementing enterprise backup is a lot like retrofitting an old building with AC. You touch everything, you trigger old problems and find new ones, and you're forced to integrate with equipment and systems you haven't seen in years-or ever. Experience is key.

"Once they installed the system, I identified some performance bottlenecks which, in an enterprise backup system, could be anywhere in the network," recalls Atwood. Intelligent Solutions pinpointed the problem, which involved products from multiple vendors-including Cisco, Compaq, and Intel-and fixed it.

Compression is key

Because of compression, PNM got even more capacity than they thought. "Each Sony DTF-2 tape has an advertised 200GB native capacity and 500GB with compression," says Atwood. "But we've achieved as much as 788GB on a tape. Out of 100 tapes, about 40 are holding more than 600GB."

As a result, PNM has gone from having four days to four full months of near-line backup data in the library on-site. "We can find and restore anything in minutes-even remotely, over an Internet connection, if we happen to be out of the office," adds Atwood. He estimates that over the next three years, the backup solution will save PNM $300,000 in unnecessary tapes, tape drives, software, and networking hardware-and that doesn't include the savings of reducing his backup staff from 10 employees to one.

Because PNM's growth projections spiked again since the backup system was installed, Atwood is especially happy with the scalability of the system. "We already added a third DTF-2 tape drive to the PetaSite library-just plugged it in, without re-engineering or increasing management duties. This kind of scalability is important to us, because the company is planning larger projects and an acquisition that will significantly increase our storage and backup requirements." Intelligent Solutions' Metcalfe adds that when the time comes, PNM can even scale its bandwidth. "They can upgrade from their current 100BaseTX connection to Gigabit Ethernet whenever they need it. That's another thing DTF does that DLT doesn't."

"With six months under our belt, we're getting faster, higher-capacity backup than we expected-and what we expected was many times what we thought was possible," says Atwood.

Mark Scapicchio is a freelance writer in the Boston area.

PNM's backup solution

Built for extreme scalability, the Public Service Company of New Mexico's enterprise backup solution includes the following components:

  • A dedicated, private backup network, built on 100BaseTX Ethernet. The network scales simply by adding switches and switch ports.
  • A dedicated Sun E3501 backup server. Intelligent Solutions chose the Unix server over NT for its superior reliability, security, manageability, scalability, and I/O throughput.
  • Sony B-150L PetaSite DTF-2 tape library. The 150-slot PetaSite library scales to a native (uncompressed) capacity of 95.6TB and a throughput of 605GB per hour (72MB per second). The PetaSite's robotics mechanism picks and places tapes in five seconds on average. The library (and the dedicated network) can be upgraded to Fibre Channel or Gigabit Ethernet whenever PNM needs the additional bandwidth.
  • Legato NetWorker backup software. Installed on the backup server-with agents on each node in the network-NetWorker lets PNM manage backups and restores from anyplace on the network (or over an Internet connection) via a Windows GUI. For fast and simple restores, NetWorker maintains a database of the contents of each tape; it also enables PNM to schedule automatic full and incremental backupss as well as tape copies for off-site storage.

This article was originally published on September 01, 2001