CONNECTIVITY: Prepare for the shift to switched fabrics

The road from PCI's shared-bus architecture to InfiniBand switched fabric architecture is through PCI-X.

By Thomas Heil

Switched fabrics are a revolutionary advance in server architecture. While new to certain markets, this architecture has existed in mainstream enterprise computing for many years. The concepts of switched fabrics are embodied in the current InfiniBand (formerly System I/O) initiative, which reflects the best of the Future I/O (FIO) and Next Generation I/O (NGIO) standards. The proposed standard promises to make systems and I/O attachments simpler, cheaper, more reliable, and easier to expand and service.

However, the transition from shared-bus architectures to switched fabrics will take time. PCI is the dominant shared-bus standard and is well established. For switched fabrics to emerge, the industry must allow a graceful transition, implying a potentially long period during which buses and fabrics will coexist. PCI-X-an enhancement to the PCI bus that is expected to emerge next year-ensures the ongoing viability of the PCI bus architecture and provides a non-disruptive path from bus-based systems to fabric-based systems.

What is switched fabric I/O?

In contrast to shared buses such as PCI, switched fabrics connect processor and I/O nodes via an "any-to-any" network of switches and point-to-point links (see figure). Nodes communicate using intelligent messaging and protected remote DMA (RDMA) techniques, in contrast to bus-based operations in and out of a shared memory.


Diagram shows one estimate of the time shift from Bus-Only to Fabric-Only architectures.
Click here to enlarge image

This logical isolation and the inherent physical isolation of point-to-point links make switched fabrics inherently more robust and secure than buses, and they are inherently scalable. Additional nodes, switches, and links are easily added in run-time to increase functionality, bandwidth, and redundancy.

Switched fabric I/O standards

Switched fabrics are not new. Proprietary fabrics have existed in mainframe, massively parallel, and Unix midrange markets for years. What is new is the intent to define industry standards that enable the fabric to eventually replace the bus as the I/O interface in high-volume markets such as Intel architecture systems. Today, Fibre Channel (switched fabric and arbitrated loop topologies) is emerging as the standard interconnect for storage area networks (SANs). Ultimately, however, a system area network standard-one that can handle all I/O functions-is needed.

Switched fabrics offer the following promises:

  • A "unified" system area network, in which a common switched fabric handles all I/O functions, including cluster inter-processor communication (IPC), network, and storage.
  • A limited distance focus emphasizing "intra-chassis" or "same room" connectivity. (Fibre Channel and LAN/WAN technologies will handle long distance applications.)
  • Incorporation into host chipsets to eventually eliminate PCI slots.
  • Definition of a new "fabric I/O adapter" that plugs into a switched backplane instead of a shared bus. Fabric I/O adapters will be hot-pluggable and externally accessible.

Bus-to-fabric transition

The transition from bus-only architectures to fabrics will probably be gradual (see figure on next page). The Bus-Only curve represents the transition starting point-i.e., servers with PCI slots only. A Bus-Only server will connect to a fabric with a PCI-SIO host adapter. The Bus-Fabric Hybrids curve represents servers with PCI slots and native (integrated) fabric channels. The Fabric-Only curve represents the transition end-point-i.e., servers with no PCI slots where all I/O occurs over native fabric channels.

Bus-Only systems will dominate the early years, while switched fabrics (implemented via PCI adapters) are introduced, debugged, and proven in production environments in all the major operating systems. During this time, a peripheral base and support infrastructure will emerge. Although some early-adopter OEMs may provide native fabric channels sooner, the broader market is typically reluctant to integrate technology that is not thoroughly proven or is not widely available. PCI slots are a low-risk proving ground.

Once fabric technology is proven and a reasonable peripheral base exists, OEMs can increasingly ship Bus-Fabric Hybrids. Over a period of several years, OEMs will shift the mix from "mostly PCI" to "mostly fabric" as the fabric peripheral base grows (similar to the way OEMs used PCI-ISA hybrids to move the ISA adapter base to PCI). This process will continue until PCI connectivity is no longer required.

Diagram shows one estimate of the time shift from Bus-Only to Fabric-Only architectures.
Click here to enlarge image

Note the phase overlap in the figure. This reflects different adoption characteristics in different segments. For example, Fabric-Only systems may emerge quite early-as soon as the technology is proven-in high-density rack mount applications. In this environment, PCI slots may be especially problematic, and access to PCI adapters may be provided through an "adapter pool" server elsewhere in the rack. In contrast, Bus-Only systems may persist for many years in small, cost-sensitive pedestal servers and PC workstations. This segment may be unable to bear the cost of both PCI slots and native fabric channels, but cannot switch until all of today's PCI functions are available in fabric form. As long as this high-volume segment remains PCI-centric, it will fuel ongoing PCI investment.

PCI-X provides a bridge

PCI-X is a significant step in the evolution of PCI. It increases bandwidth with higher frequencies and protocol enhancements as well as scalability with electrical improvements that enable OEMs to provide more high-frequency slots. PCI-X and other initiatives like hot-plug and alternative form factors will significantly extend the useful life of the bus architecture while maintaining legacy compatibility. Switched fabrics are compelling, but it's clear that bus architectures will play a critical role well into the next century, and PCI-X provides an evolutionary path.

As switched fabrics emerge, the role of Fibre Channel and SCSI may change somewhat. Today's PCI-SCSI and PCI-FC adapters will gradually give way to fabric adapters. These adapters will enable server vendors to replace PCI slots with native fabric channels, while providing connectivity to Fibre Channel SANs and SCSI or Fibre Channel peripherals.

Switched fabric I/O is the architecture of the future, with potential to deliver significant value to end users. It is, however, a fundamental architectural shift that will unfold over many years. Just as important to end users is a non-disruptive path to this new architecture, which implies a period of bus/fabric coexistence.

Thomas Heil is a senior systems architect at LSI Logic ( invalid link: www.lsilogic.com www.lsilogic.com ), in Milpitas, CA.

This article was originally published on January 01, 2000