CS3 is a financial industry leader migrating to an IT model that relies on a “flat” or horizontal server approach, meaning it uses many smaller servers rather than smaller numbers of large servers. This proves challenging for the storage group because CS3 is often rolling in dozens of new servers each week. The company’s storage group struggled to continuously provide adequate resources for application users and turned to storage virtualization and thin provisioning to help solve the problem.

Pain points
CS3 was having difficulty managing rapid server growth. It was embarking on a development project for a new financial trading system that was generating high demand for storage capacity and server support. The result was the following:

  • The company was unable to provide adequate support for the rapid growth of new servers, often installing dozens of additional servers per week;
  • CS3 was constantly running out of storage and having to purchase more storage every few months to meet demand; and
  • The company was constantly running short of other critical resources, including switch ports and storage ports.

As such, the firm’s practice was to over-provision storage to allow for headroom. The executive in charge of storage operations said, “It was common for us to provision 400GB to a server when its applications really only needed 50GB.”

Solution strategy
CS3’s current production infrastructure consisted primarily of EMC DMX-3 arrays, but the company wanted to find a second source for its production systems. New applications were being developed on Sun servers; however, as a generally high-end EMC storage shop, the company was at first not disposed to use Sun’s midrange storage arrays for these applications. CS3 began to evaluate the Hitachi Universal Storage Platform (USP) V array, which is sold through Sun, and was attracted to its virtualization, virtual ports, and thin- provisioning capabilities.

CS3 is about nine months into the project. In the first two months, the company received management approval to start evaluating the capabilities of the USP V and has since rolled out the solution for its test and development environment.

The Sun/Hitachi array supports virtualization and thin provisioning for both internal arrays and externally attached arrays from a variety of vendors. CS3 is currently using only the internal virtualization and thin-provisioning (“Dynamic Provisioning”) capabilities of the USP V and has begun to evaluate both of these features for externally attached storage.

Adoption issues
The biggest adoption issue for CS3 was the migration from its existing DMX-3 environment to the Sun/ Hitachi USP V approach. While EMC infrastructure remains the mainstay of the production systems, the move to new storage arrays required substantial planning and careful migration, which was nevertheless disruptive.

CS3 needed to learn some new concepts related to virtualization and thin provisioning. The storage team used the GUI that came with the Sun/Hitachi array to manage the virtualization and set up resources such as port allocations.

Another adoption issue was related to procurement practices and thin provisioning. Because thin provisioning essentially “fakes” the application into believing it has more storage than is physically available (avoiding over-allocation), when thresholds were met and CS3 had to acquire additional storage, previous procurement practices were exposed as too lengthy. It quickly became clear the procurement process was problematic and needed to be improved or applications would be at risk of crashing.

CS3 put in place a new streamlined procurement process to accelerate the approval of storage purchase orders.

The project is nine months in to a multi-year plan to extend storage and server virtualization across the organization. Currently, the scope of virtualization and thin provisioning is confined to test and development. The approach allowed CS3’s test and development organization to use less expensive servers, which served its horizontal server strategy. The key benefit of the virtualized storage infrastructure was the ability to easily and quickly provide storage and connectivity to a rapidly growing server farm.

Despite the substantial initial migration and setup costs, a significant benefit for CS3 was that thin provisioning allowed the organization to put off purchasing an entire array, estimated at more than 40TB, which translated into a net savings of several hundred thousand dollars.

CS3 had five production servers connected to EMC DMX-3 arrays that were growing rapidly (more than 300TB per year). The development and test environment closely mirrored this scenario, but the absolute number of terabytes was much smaller. CS3 decided to deploy many smaller Sun and IBM Windows servers using the 40TB Sun/Hitachi USP V array to support the test/development infrastructure. Because the test and development servers did not have the same performance requirements as the production environment, CS3 was confident that it could match the storage and server requirements with no performance issues. Such was the case as CS3 has had no performance problems and believes the Sun/Hitachi arrays could be candidates for the production environment in the future. However, the company has not done adequate testing to make this final determination.

CS3 chose to only use virtualization on 40% of the Sun/Hitachi internal storage array. The company applied thin provisioning to the virtualized portion of the array by spreading data in very small chunks across a wide span of devices, creating one large pool of storage. CS3 was immediately able to reclaim wasted space-as much as 300GB per server across dozens of servers. In total, over a nine-month period CS3 estimates it recaptured approximately 40TB of over- allocated storage, allowing it to defer a previously planned array acquisition.

Organizational issues
The organizational skills’ change that occurred as a result of this project was the need to put in place new policies related to the physical amount of storage being allocated, versus the amount actually being used. CS3 experienced a two- to three-month learning curve, working with Sun to fine-tune the parameters and monitor the system behavior via the GUI. CS3 would run performance reports off the USP V every night and send it to Sun for analysis and recommendations.

Once CS3 was comfortable with the behavior of the system it turned its attention to the new business process related to procurement. Thin provisioning exposed a shortcoming in the procurement process whereby when storage pools run low, then physical storage needs to be added immediately and delays in a purchase order process can be critical. CS3 implemented an approach that reduced the number of sign-offs required and allowed for multiple signatories in the event key people were out of the office. While seemingly trivial, this change was crucial.

Best practice
CS3’s advice to other companies is to use virtualization and thin provisioning as a technique to improve utilization and simplify migration efforts. However, there are initial costs of migrating from existing infrastructure that must be considered. Specifically, for a medium-to-large installation, in addition to initial capital costs for array hardware and software, costs associated with incremental software licenses, services, and migration from existing arrays could exceed $200,000.

To offset these costs there must be enough change of server and storage resources to justify the expenditure. To the degree that this is the case, saving several hundred thousand dollars (as was the case with CS3) within a year is feasible. This move could also save organizations tens of thousands of dollars for each array migrated.

For companies that have arrays rotating on three-year leases, for example, this can bring significant savings once the initial migration is completed.

The other advice CS3 offers is to limit the complexity of storage resource management (SRM) tools. CS3 does not believe in an umbrella-based SRM approach and recommends mastering the native tools that come with storage systems. While this puts extra load on the storage administrators, CS3 believes mastering native tools is ultimately more effective than trying to achieve a one-tool-does-all strategy.

CS3’s use of storage virtualization and thin provisioning targeted at its test and development environment is prudent and underscores the immaturity of the technology in general, which is why the company started in a less performance-intensive environment to gain experience with these promising technologies.

A practical focus on reclaiming wasted storage space enabled CS3 to demonstrate quantifiable return on investment to upper management. Moreover, once the initial migration to the virtualized arrays was completed, CS3 benefited from more rapid addition of new capacity and migration of existing arrays and servers, non-disruptively.

Nonetheless, CS3 had to overcome some initial migration disruption and other costs associated with moving to a virtualized, thinly provisioned environment.

Despite some initial concerns about the performance and reliability of the approach, CS3 reports very positive results.

Going forward, CS3 has the opportunity to extend virtualization and thin provisioning to external capacity attached to the USP V. This is forthcoming and will further support the integration of an increasingly diversified installed base of arrays.

Moreover, as CS3 begins to implement server virtualization, the pace of server change will escalate and the company will be well-positioned to respond.

It is unclear at this time if CS3 will drive this strategy to the production environment. Concerns about the performance and novelty of thin-provisioned volumes loom large, and the company is biding its time to gain more experience with storage virtualization and thin provisioning.