SPC releases 'standard' benchmark for disk arrays


The Storage Performance Council (SPC) recently announced the SPC-1 benchmark, a standardized test to measure the performance of disk subsystems. According to Leah Schoeb, SPC chairperson and a staff engineer at Sun, the benchmark is designed for both disk array manufacturers and end users.

Manufacturers can use the results of the tests to compare the performance of their arrays to that of their competitors, while end users can use this information to make it easier to develop a "short list" of prospective suppliers. Schoeb says that IT organizations may also use the SPC-1 benchmarks to specify performance parameters in their RFPs.

The suite was developed by SPC members Adaptec, Compaq, Dell, The Evaluator Group, Hitachi, IBM, Ideas International, LSI Logic Storage Systems, NEC, Sun, Unisys, and Veritas Software. Conspicuously absent from that list are EMC and Hewlett-Packard, both of which were originally members but subsequently dropped out.

The SPC-1 benchmark has two types of tests and metrics. SPC-1 IOPS (I/Os per second) measures the maximum I/O-request throughput of a storage subsystem in a system environment. The SPC-1 LRT (Least Response Time) test measures I/O-request response times in an application environment. Business Scaling Units (BSUs) represent the I/O load imposed by an application's users.
Click here to enlarge image

As evidence of how difficult it is for such a diverse group of vendors to develop a standard benchmark, the SPC was formed in 1996 and SPC-1 is the group's first deliverable. The benchmark was originally due a year ago.

The SPC process is modeled after the Transaction Processing Council and its benchmarks. Manufacturers first run the tests, and then the results are certified by an independent auditor and sent out for a 60-day peer-review process. In the case of disputes, an SPC technical advisory board, consisting of five individuals, will arbitrate. Benchmark results will be posted on the SPC Website (www.storageperformance.org), as well as on the Websites of the SPC's "preferred analyst" firms-The Evaluator Group and Ideas International. Schoeb expects the first benchmark results to be posted in the next month or two.

The benchmark emulates random-I/O OLTP, Web server, data base and e-mail server environments, and data streams and is designed to test performance and determine price-performance ratios. It is not designed to measure features such as reliability, availability, and serviceability. Basically, the benchmark plots response time against throughput (see chart). Tests can be conducted in direct-attached or network storage configurations.

Although the initial benchmark is for OLTP-type applications, in the future, the SPC plans to add benchmarks (including one for network-attached storage devices), as well as the ability for end users to vary the workload of the test to more accurately simulate their own applications.

The SPC is a non-profit organization funded by its member companies. For more information, visit www.storageperformance.org, or call (650) 556-9384.


This article was originally published on January 01, 2002