Tech preview: Serial-Attached SCSI + PCI Express

Later this year you will be able to combine Serial-Attached SCSI and PCI Express to maximize server-storage performance.

By Paul Griffith

Tight IT budgets are increasing pressure on organizations to use system resources with greater efficiency to serve more clients. More than ever, IT managers must choose technologies that make the most efficient use of their server and storage components as the amount of information they manage continues to grow. One system resource that has come under increasing scrutiny is the storage system. These systems are at the heart of a data center's ability to make data available 24×7, seamlessly scale capacity, and increase server-storage performance while reducing management costs.

For more than 20 years, data centers have relied on parallel bus interfaces as the workhorse disk drive interconnects. Parallel technologies remain in widespread use and continue to meet the performance requirements of many enterprise applications. However, significant technical challenges have made it economically impractical to use parallel technologies to meet increasing demand for higher-performance, more-robust data integrity, greater storage flexibility and scalability, and smaller system designs.

Serial storage technology, specifically Serial ATA (SATA), Serial-Attached SCSI (SAS), and PCI Express, has emerged to address the architectural limitations of parallel technology to deliver highly scalable performance. The technology draws its name from the way it transmits signals—in a single stream, or serially, compared to multiple streams for parallel. The main advantage of serial technology is that while it moves data in a single stream, it wraps data bits into individual packets that are transferred up to 30 times faster than with parallel technology. In addition, serial technology's point-to-point architecture features dedicated connections that deliver full bandwidth to each device.

Serial architectures also deliver higher performance by allowing more bandwidth per device pathway than their parallel counterparts. Serial architecture connections consist of a single pair of transmission signals that contain an embedded clock for self-clocking, enabling clock speed to be easily scaled. Serial bus architectures also support a network of dedicated point-to-point device connections, versus the multi-drop architectures of parallel buses, to deliver full bandwidth to each device, eliminate the need for bus arbitration, reduce latency, and simplify hot-plug and hot-swap system implementations. A dedicated serial connection also eliminates the single point of failure found in today's parallel environments.

Serial ATA

Serial ATA, which is shipping today, extends the parallel ATA technology road map by delivering disk interconnect speeds starting at 1.5Gbps (150MBps). Due to its lower cost-per-gigabyte, SATA will continue to be the prevalent disk interface in PCs, entry-level servers, and entry-level networked storage systems where cost is a primary concern.

Serial-Attached SCSI

Serial-Attached SCSI, the successor to the parallel SCSI interface, will leverage proven SCSI functionality and features while expanding performance, scalability, and reliability for enterprise storage. SAS will offer many new features such as drive addressability up to 16,000 devices per port and reliable point-to-point serial connections at first-generation speeds up to 3Gbps. In addition, SAS's small connector supports full dual-ported connections on 2.5-inch hard disk drives, a feature previously found only on larger 3.5-inch Fibre Channel drives. Dual-ported connections are essential for applications that require redundant drive spindles in a dense server form factor such as blade servers.

The SAS interface will also be compatible with lower cost-per-gigabit Serial ATA drives, giving system builders the flexibility to integrate either SAS or SATA devices and substantially reduce procurement, inventory, and other costs associated with supporting two separate interfaces.

Serial-Attached SCSI will improve drive addressability and connectivity using an expander that enables one or more SAS host controllers to connect up to 128 ports, which may include other host connections, other SAS expanders for even greater scalability, or hard disk drives. Connecting multiple expanders together will achieve connectivity to more than 16,000 devices. This highly scalable connection scheme enables enterprise-level topologies that easily support multi-node clustering for automatic fail-over availability or load balancing.

PCI Express, a new serial host interconnect architecture, is designed to address future system interconnect requirements by delivering the flexibility, scalability, and performance needed to support upcoming technologies like 10Gbps Ethernet and SAS. A point-to-point architecture with hot-plug and hot-swap support, PCI Express is software-compatible with PCI and PCI-X to simplify the design of next-generation serial systems.

Figure 1: Multiple PHYs can be combined to form wide ports that support the bandwidth requirements of large Serial-Attached SCSI topologies.
Click here to enlarge image

PCI Express uses a dual simplex serial data stream with an embedded clock to overcome many of the performance limitations of parallel bus architectures. A PCI Express link consists of two low-voltage, differentially driven pairs of signals: a transmit pair and a receive pair. And like SAS, a data clock is embedded using the 8b/10b encoding scheme to achieve very high data rates. Each point-to-point interconnect may have 1, 2, 4, 8, 12, 16, or 32 dual simplex 2.5Gbps lanes (2Gbps effective rate), providing scalable bandwidth up to 12Gbps (16GBps) between nodes. In comparison, a typical 64-bit, 133MHz PCI-X 1.0 device provides approximately 1GBps bandwidth.

Wide Serial-Attached SCSI ports

First-generation Serial-Attached SCSI will deliver throughput of 3Gbps per link and succeeding generations up to 12Gbps to keep pace with technology and application advances. In addition, SAS's full-duplex, point-to-point architecture will support simultaneously active connections among multiple initiators and high-performance SAS targets. Devices can transfer data in both directions at once to effectively double the usable bandwidth of the link rate. These multiple links, in turn, can be combined into wide ports, allowing system designers to aggregate the performance of SAS initiators and expanders to increase total available bandwidth. Grouping four or eight links can produce bandwidth of 12Gbps or 24Gbps, respectively (see Figure 1 on p. 26).

Serial-Attached SCSI expanders

The scalability of parallel buses is limited because they share connection paths, and adding more buses with multiple initiators does little to extend this limited sharing ability. Serial-Attached SCSI uses expander hardware as a switch to simplify configuration of large external storage systems that can be easily scaled with minimal latency while preserving bandwidth for increased workloads. Expander hardware enables flexible storage topologies of up to 16,256 mixed SAS and Serial ATA drives.

Figure 2: Expanders enable the design of very large storage topologies. Each fan-out expander can be connected to up to 128 physical devices per PHY, including multiple initiators and other edge expanders.
Click here to enlarge image

For example, one type of expander, a fan out, can connect up to 128 devices per PHY, including initiators, SAS and SATA drives, and edge expanders with either narrow or wide links (see Figure 2). These additional initiators and edge expanders can in turn be linked to other hosts and drives, providing additional connection nodes. The SCSI Management Protocol (SMP) within SAS manages the point-to-point connections in the topology.

Serial-Attached SCSI bandwidth

First-generation Serial-Attached SCSI link rate is 3Gbps (300MBps) and supports full-duplex data transfers for up to 600MBps bandwidth. And because SCSI protocols are not restricted to half-duplex operation, SAS will also support full-duplex transfers, allowing data to be transferred simultaneously in both directions to maximize bandwidth. For example, a device can simultaneously transfer data from a previously queued read operation while receiving data for a write operation. Although full duplex will not be used during all transfers, this feature can double the usable bandwidth of the link rate. In contrast, Ultra320 SCSI's shared-bus architecture is restricted to 320MBps for all attached devices. SAS's support for wide ports further improves throughput by enabling several disks to communicate with a single port address simultaneously.

For example, a SAS controller with four 3Gbps links configured as a wide port will support data-transfer rates of 1,200Gbps or 1.2GBps at half duplex. A SAS controller configured with eight links supports data rates of 2.4GBps or 2,400MBps at half duplex.

Currently, a 15,000rpm disk drive will sustain data rates up to 75MBps. At these data rates, two disk drives will saturate a 1.5GBps Serial ATA bus. The shared Ultra320 SCSI bus supports a total of 320MBps or the sustained data rates of four to five disk drives. By contrast, a 4-wide SAS port supports as many as 16 hard drives before becoming saturated (see Figure 3).

Figure 3: By combining links, Serial-Attached SCSI supplies the bandwidth needed to support the sustained data rate of many attached disk drives. Assuming each drive can sustain a data rate of 75MBps, a SAS topology with a 4-wide port could support up to 16 disk drives before throughput saturation.
Click here to enlarge image

Serial-Attached SCSI's ability to aggregate bandwidth through the use of wide ports will support the performance scalability required by next-generation servers and storage systems. However, while SAS can supply the bandwidth for next-generation storage I/O, it requires a proficient host interconnect to optimize total system performance.

SAS and PCI Express

System architects typically optimize performance by eliminating bandwidth bottlenecks—a goal typically met by matching interleaving technologies with complementary efficiency and availability levels.

Like SAS, PCI Express delivers scalable performance by combining multiple data links to create wide data paths. This common capability is the key to optimizing performance between SAS and PCI Express.

Figure 4: A 4-wide Serial-Attached SCSI port supports bandwidth of 1,200MBps, enough for up to 16 attached drives. A SAS host attached to a PCI Express 8-wide lane, which can accommodate 2,048MBps, delivers the system bandwidth to support this streaming data rate.
Click here to enlarge image

By combining SAS with PCI Express, system designers can generate bandwidth for at least 16 hard disk drives, with neither technology bottlenecking the performance of the other. With today's drives generating sustained data rates of about 75MBps, 16 drives will require about 1,200MBps bandwidth. A SAS port configured as 4-wide supports 1,200MBps while a PCI Express slot configured for 8-wide supports 2,048MBps (see Figure 4).


As data centers are called on to serve more clients, IT managers must choose technologies that optimize the capabilities of their server and storage components. Deploying complementary components within the system is a vital step in that direction.

New serial technologies are emerging to overcome the bandwidth limitations of today's parallel architectures and deliver highly scalable performance for next-generation systems. Serial-Attached SCSI delivers a 3Gbps per link data-transfer rate, the flexibility to deploy either low cost-per-gigabit Serial ATA drives or high-performance SAS drives in the same system, point-to-point connections for high reliability, and highly scalable connectivity to more than 16,000 devices in a single domain. PCI Express offers similar benefits, with each PCI Express lane supporting 2.5Gbps performance with scalability up to a 32-wide lane configuration.

By combining SAS's wide ports with PCI Express's wide lanes, system designers can maximize total storage-system performance. For example, a 4-wide SAS port will support bandwidth of up to 1,200Gbps for up to 16 hard disk drives. And with the SAS initiator attached to a PCI Express 8-wide lane, the entire data path through the system will be optimized with bandwidth to spare.

For more information about SAS, visit the SCSI Trade Association at www.scsita.org.

Paul Griffith is a strategic marketing manager at Adaptec (www.adaptec.com) in Milpitas, CA.

This article was originally published on June 01, 2004