Case Studies: Storage Consolidation part 2
By Michele Hope
Storage consolidation projects often begin as a means to reduce the storage management burden and gain more-efficient control of data growth and data protection. This was the case with Atlanta-based Cbeyond, a voice and broadband Internet provider for small businesses.
Cbeyond has approximately 15,000 customers that use the company’s Voice over IP (VoIP) and soft-switch technologies for services such as local and long-distance calling, high-speed T1 Internet access, e-mail, voicemail, conferencing, VPN, and Web hosting.
According to Allen Thomas, senior director of systems operations, Cbeyond’s growth resulted in an IT environment with about 200 dispersed servers with various operating systems that support Cbeyond’s core data services, a variety of online support systems, and a slew of back-end databases.
With about 10TB of mostly direct-attached storage (DAS), Thomas described the arrangement as “a vast collection of lots of local drives buried in servers, a number of direct-attached arrays, and a handful of Network Appliance NFS servers.”
With data growing at about 6% per month, Thomas and his seven-person IT team recognized that Cbeyond would need storage systems that could keep pace with the company’s growth rate.
In 2003, the IT team began to consolidate the company’s DAS to a SAN. After evaluating a number of solutions from large storage vendors, Cbeyond opted for two 3PAR InServ Storage Servers, each with 9TB of raw capacity.
A primary InServ Storage Server was installed at Cbeyond’s Atlanta production environment, with a backup array at the company’s off-site disaster-recovery facility in Dallas.
One benefit that Cbeyond gained from consolidation is that backups now take a fraction of the time that they used to take. Using 3PAR’s point-in-time copy capabilities, Thomas’ team now schedules frequent snapshots or disk images of primary server data throughout the day. It also backs up much of the primary server data to the secondary array, with tape backups shipped nightly to Dallas for disaster-recovery purposes.
Thomas applauds 3PAR’s approach to provisioning. “You’re not pre-allocating all your data. You only allocate storage when you write to it, which was huge for us,” he says. The “thin-provisioning” functionality comes from 3PAR’s “dedicate-on-write” technology, where capacity is allocated only when an application writes to it. This is in contrast to what 3PAR calls “dedicate-on-allocation” approaches, which can make storage capacity potentially inaccessible to other storage volumes that might need it.
“Provisioning is now trivial,” says Thomas. “You can provision terabytes of data in minutes, and creating three 2TB partitions takes 15 seconds.”
Today, Cbeyond multi-paths its critical servers into the InServ array. Each server has two independent Fibre Channel connections going directly to two independent controllers in the 3PAR array. For now, the company is not using a switched fabric, although it may move to a Brocade SAN as less-critical servers come online.
How much has the move to consolidated, networked storage helped solve Cbeyond’s storage troubles? The IT team now spends less than 3% of its time managing its storage.
Consolidation through virtualization
When network services manager Montie Massengale first joined the Las Vegas-based network of Valley Health Systems hospitals, he saw what he thought was a fairly typical IT shop. The network of four regional hospitals had about 80 servers (mostly Windows NT), each with DAS.
The problem with this configuration was that Massengale and his team of 18 frequently had to buy additional servers to accommodate the needs of new applications, storage bays filled to capacity, or saturated servers.
This growing server count not only proved costly and inefficient, but it also put a significant strain on IT’s ability to adequately back up and protect critical data. With one DLT tape drive attached to each server and Computer Associates’ ArcServe 6 on the network, Massengale’s team could spend from 12 to 15 hours or more backing up all the servers on a routine basis. And the backup infrastructure didn’t support the need to rapidly restore files.
One of the company’s initial goals was to implement a new backup system capable of providing users with rapid restore of files and data. Massengale tapped storage reseller and integrator West Coast Technology, in Aliso Viejo, CA, to help solve the data-protection problem and move the organization away from its reliance on DAS.
Massengale wanted to use virtual servers, a SAN with a virtual pool of storage, and data mirroring from one site to another in case a disaster brought down IT systems at one of the hospitals. He also wanted to replicate Valley Health’s IT data back to a Hewlett-Packard SAN at its parent company, Universal Health Systems, in King of Prussia, PA.
The first step on Massengale’s road to consolidation was to upgrade the underlying network to support a more robust networked storage infrastructure. This involved laying down optical fiber between the four hospitals in order to obtain 600MB of bandwidth across a WAN.
Massengale saw this as a foundational step required to support future consolidation projects. “The core of everything was the implementation of the WAN,” he says. “You couldn’t do what I had in mind over T1s. Moving to fiber gave me the bandwidth I needed to implement the enterprise storage system, SANs, and mirroring.”
To help solve Massengale’s immediate backup issues, West Coast Technology recommended a backup appliance from STORServer, which included Tivoli Storage Manager software and an Overland Storage tape library. He liked the appliance’s archiving function and its ability to allow end users to restore their own files, as well as the fact that it integrated well with the SAN.
West Coast suggested a handful of SAN vendors, but StoreAge Networking Technologies quickly rose to the top of Massengale’s list in terms of price and performance with its virtual SAN management appliance, StoreAge Virtual Manager (SVM), which StoreAge had coupled with Nexsan ATAboy and ATABeast disk arrays and QLogic Fibre Channel switches and HBAs.
Massengale particularly liked StoreAge’s integrated software, including software that provides mirroring data across multiple sites. StoreAge software includes multiView, multiCopy, and multiMirror. “With a lot of the other vendors that functionality comes at an extra price,” says Massengale.
Designed to allow the ability to manage virtual storage volumes and LUNs from a heterogeneous mix of storage hardware, StoreAge’s solution also enabled integration with the remote HP SAN. Massengale also liked the fact that he could upgrade his SAN to include newer, less-expensive disk technologies, such as Serial ATA, as they came out.
Massengale implemented the StoreAge solution in a “hub-and-spokes” network configuration-with 3.5TB of ATA storage at the hub hospital to support mirroring data from the other sites, and 1.75TB at each hospital to support primary data needs. To ensure no single point of failure, Valley Health incorporated a dual-fabric, Fibre Channel SAN with QLogic switches and adapters to connect the Nexsan disk arrays to the servers.
Now, hospital data is routinely backed up and mirrored to the hub site. Valley Health also uses StoreAge’s “boot-from-SAN” capability to enable 20-minute restores of critical servers in the event that they go offline.
RAID-0 configurations on disks associated with less-critical servers provide disk mirroring in the event of local hardware failures, but Massengale’s ultimate disaster-recovery fallback is mirroring data across the WAN. “In every instance we still mirror the data off to another site,” he says, “so if anything happens to the SAN we still have that data at another site and can access it.”
As opposed to more-traditional storage options, such as optical jukeboxes, the SAN is a cost-effective means to further consolidate Valley Health’s storage. Massengale estimates that the cost of a 17TB SAN with RAID arrays is about one-third that of many jukebox-based systems.
Today, Valley Health’s mirrored SAN architecture is being used as a blueprint for other hospitals to follow in the Universal Health Systems network. Two hospitals in Florida have also implemented a StoreAge SAN configuration, using iSCSI instead of Fibre Channel.
NAS for storage consolidation
Sometimes, timing plays a crucial role in an organization’s decision to undergo widespread storage consolidation. At San Diego-based Accelrys, a computational science company, the IT group faced a convergence of events that required it to implement a more consolidated networked storage architecture.
Accelrys’ main IT operations support more than 580 employees and span three centers in San Diego, Cambridge, UK, and Bangalore, India. At each of these centers, the IT team develops and supports a range of informatics and computation-oriented software products and services used by pharmaceutical, biotechnology, and chemical companies.
The events leading to Accelrys’ search for a consolidated storage solution included the fact that several of the company’s Wintel servers, already experiencing their share of disk problems, were fast approaching their end of life. The company also had some Network Appliance NAS filers that were at maximum capacity and could only be upgraded by migrating to larger filers. Rob Butterworth, Accelrys’ IT director in Cambridge, says this would have required a large investment and full-scale replacement of Accelrys’ existing filers.
The IT group was also facing mounting requests from users to increase disk capacity for local disks on various workstations. Employees stored large amounts of data on their workstations, which made it difficult for the IT group to institute comprehensive data-protection, backup, and disaster-recovery policies.
In an effort to tackle the local disk storage issue, the IT group had been forced to develop its own software processes to back up the 2TB+ of local data. “The localized storage on workstations was a big issue for us,” says IT manager Carol Busch. “One of the main issues was trying to quantify the amount of data stored.”
According to Butterworth, the issue of locally attached storage also presented some extra challenges for Accelrys’ IT staff, which consisted of six people in San Diego, six in Cambridge, and two in Bangalore. Such challenges were another impetus for consolidating their storage.
“We had a large number of machines with local storage. We were concerned about knowing where the data was, the fact that critical data was stored in different places, and about backing up all the data,” says Butterworth.
In San Diego, Busch and her team had been supporting the data storage needs of about six Windows and Unix-based file servers, along with several dozen work-stations, with approximately 11TB of data. IT staff in Cambridge had to support an additional 6TB to 8TB of data.
When Accelrys began looking at networked storage solutions, it considered both NAS and SAN. “We concluded that SANs didn’t give us much more performance over a NAS solution,” says Butterworth, “and we couldn’t justify the initial cost, in terms of capital and support, of a SAN.”
The support issue was a critical factor in Accelrys’ ultimate storage solution, according to Busch. With no dedicated IT staff for storage-related tasks, just about any staff member might be called upon to manage storage functions related to the new solution.
Easier administration was at the top of the priority list, along with performance, scalability, and handling of file-based security access controls. Down the road, the IT group also wanted to implement more granular hierarchical storage management (HSM) policies to migrate data from primary to secondary storage.
Although they initially wanted the solution to improve backup performance, they also needed a more robust disaster-recovery strategy that would allow for cross-site backups and access to key data from other sites.
Accelrys ultimately acquired three Titan SiliconServers from BlueArc, one for each of its centers. “BlueArc’s approach was a completely different way of handling storage,” says Busch, “and the systems scaled without having to be replaced.”
Each of the Titan SiliconServer arrays uses a mix of Fibre Channel and ATA drives, a feature BlueArc calls “multi-tiered storage,” which fit in well with Accelrys’ HSM goals.
Once the migration is complete, Butterworth expects that managing the system will require only half the amount of IT time it used to take. “We don’t have a lot of different storage systems with different vendors providing management tools.”
Accelrys has implemented an 8.1TB Titan array in Cambridge and a 5.8TB system in Bangalore and plans to implement a larger 18.6TB array at its San Diego location.
Before embarking on a storage consolidation project, Butterworth advises companies to upgrade their networks. “You can’t do high-performance, reliable NAS if your network isn’t up to the challenge.”
Michele Hope is a freelance writer and owner of TheStorageWriter.com. She can be reached at firstname.lastname@example.org.