Inside the SPC-1 benchmark

By Leah Schoeb

The SPC Benchmark-1 (SPC-1), a standard performance measurement workload for enterprise network storage, produces both performance and price/performance results.

SPC-1 is implemented in a test kit called a Workload Generator that stresses a storage subsystem to demonstrate its performance while executing the functions of common business- critical applications.

The Workload Generator creates multiple streams of I/O commands to exercise a tested storage configuration. The streams represent a variety of I/O access patterns found in real production environments. The access patterns include random, sequential, and random walk with hierarchical re-use. These patterns also make consideration for real-world read/write ratios, arrival distributions, data locality, and transfer sizes. The subsystem is divided into three Application Storage Units (ASUs) that model the real-world division of storage resources for database and e-mail-server applications. Separate I/O streams are defined for each of the three ASUs.

Each I/O stream is made up of one or more concurrently executing instances that generate I/O commands for a specific ASU. Each concurrently executing instance has a consistent and self-contained pattern of behavior, which does not depend upon other stream instances or the overall load level of the system. The I/O Stream Generator is responsible for generating a sequence of I/O requests for each instance of each ASU stream. The I/O Command Generator is responsible for submitting I/O requests in the sequence defined by the I/O Stream Generator. The I/O Command Completion module records the completion time and forwards key metrics to the Statistics Collection module.

Tests and metrics

Vendors ("test sponsors") are required to report two primary metrics when publicly describing their results:

  • SPC-1 IOPS, which is the maximum total throughput of the configuration; and
  • SPC-1 LRT, which is the best-case response time of the configuration.

Test sponsors must also disclose the capacity of the tested storage configuration (which includes resources to support data protection and redundancy schemes) as well as the price/performance of the configuration.

As an "enterprise" storage benchmark, SPC-1 requires vendors to run several test components to demonstrate data persistence, repeatability, and sustainability. Below is a summary of these tests.

Click here to enlarge image

Data Persistence-Storage used in an SPC-1 test must demonstrate the ability to preserve data without corruption or loss.

Repeatability-A benchmark configuration must demonstrate the ability to reliably reproduce SPC-1 performance results across system restarts.

Sustainability-The Sustainability Test Phase demonstrates the maximum sustainable I/O request throughput within at least a continuous three-hour measurement interval.

Response Time Ramp-This test phase is required to produce a response time/throughput curve. It measures various load levels and provides the average response time of a lightly loaded TSC (the SPC-1 LRT result).

Leah Schoeb is the chairperson of the Storage Performance Council (SPC).

This article was originally published on January 01, 2002