Inside Fibre Channel Switching

Inside Fibre Channel Switching

Fibre Channel switches form the backbone for storage-area networks, but how does Fibre Channel compare to ATM and Gigabit Ethernet?

Peter Dougherty

IT managers, once occupied with streamlining and cost-cutting measures, have turned their attention to managing the flood of information that`s critical to their organizations` bottom lines. Enterprise storage--a consolidated storage system that supports heterogeneous operating systems such as Unix and Windows NT--is one solution to the information explosion. Rather than placing information in geographically dispersed departmental servers and storage systems, IT managers can consolidate it and manage it from a central location.

Such consolidation places heavy pressure on enterprise interconnect technology. On the upside, open-systems technologies and multi-platform solutions give IT managers a wider choice of vendors and applications from which to choose, increasing protocol and architecture options and improving the overall cost efficiency of enterprise solutions. On the downside, they create a much more complicated IT infrastructure.

While other technologies (e.g., ATM and Gigabit Ethernet) have their strengths, Fibre Channel is the superior connectivity technology for server/storage backbones. Not only is Fibre Channel appropriate for open-systems environments, but it delivers mainframe-class connectivity, data availability, and manageability.


The increased connectivity of Fibre Channel switching improves information access. Fibre Channel is an ANSI standard that is protocol-independent. It supports a heterogeneous mix of servers, workstations, and storage devices. It also allows for flexible and scalable networks and has great distance capabilities--and it has the bandwidth and data integrity of channel technologies.

A Fibre Channel network eliminates the traditional point-to-point connection between a host and storage devices. Instead, any server can access any storage device or any server in the Fibre Channel fabric. Storage systems can be logically consolidated; they don`t have to be located next to each other and they don`t have to occupy expensive data-center floor space. Users gain access to new levels of information from all areas of the organization.

In multi-point configurations, storage and servers can be mirrored. If a device fails or needs to be reconfigured, administrators can assign another device on the network to take over, and business continues without interruption. Administrators can take servers off-line for maintenance, while ensuring complete storage availability to all users.

Data Availability

With Fibre Channel switching, multiple servers are interconnected to multiple storage devices, giving users complete access to centrally managed data. As servers and I/O subsystems are added, Fibre Channel is able to balance access to the data.

In a data center, path redundancy greatly affects data availability and reliability. Unlike traditional SCSI implementations that require one or more physically separate SCSI buses for each host system, a Fibre Channel switch provides a reliable network that simultaneously supports access to multiple SCSI buses.

The transfer latency of server-to-storage messaging can also be a key factor in server/storage backbone performance. With total message latency measured in tens of microseconds, the interconnect fabric must provide consistently low frame latency. Frame latency, defined here as the length of time it takes for the first bit of the frame to enter and then exit the switch, can be as low as one microsecond in a Fibre Channel switch.

"Connectionless" Fibre Channel allows concurrent multiplexing of many simultaneous SCSI and IP transfers onto a single link. By using connectionless frame switching in combination with high aggregate bandwidth and credit-based control flow, switched Fibre Channel fabrics avoid dropping frames due to high access contention.

Enterprise data centers require reliability and data integrity characteristics that meet 24x7 expectations. Fibre Channel offers a highly reliable, built-in protocol with embedded cyclic redundancy check (CRC), which, when combined with such high-availability features as component redundancy and on-line replacement, meets IT managers` expectations for an efficient, highly reliable enterprise fabric. All of these attributes contribute to exceptional data integrity.


A Fibre Channel fabric provides key management services: device and address discovery, partitioning and security, and performance and error monitoring. Fibre Channel switches provide standards-based management services that assist in device and address configuration, including

-An embedded ARP server that provides IP address resolution.

-A name service that allows the server to query the fabric for a list of SCSI targets on the fabric.

-A notification service that automatically notifies nodes when the fabric topology changes.

-Embedded SNMP master agents that enable configuration management and monitoring.

An extraordinary amount of mission-critical traffic flows through enterprise networks. The Fibre Channel standard gives manufacturers the option of providing administrators with critical network monitoring tools to determine "hot spots" and unusual traffic patterns. Equipped with these tools, administrators should be able to quickly detect and isolate paths or nodes that cause high error ratios. Fibre Channel switches can provide a rich set of management objects for monitoring at the path, link, and frame level and for allowing standard management tools to capture information via SNMP or Web-based interfaces.

For high-availability systems, switch manufacturers can provide advanced component failure detection and reporting. If a failure occurs within the switch or in the fabric, the nature and location of the problem are reported immediately, avoiding expensive delays. Some Fibre Channel switches also have management capabilities that make the systems easier to use and ultimately help ensure data availability. For example, the switched fabric could be managed through a locally attached console or over the Internet through a remote console. Status monitoring, user-configurable failover criteria, and nondisruptive service should also be functions of the management software.

Options and Tradeoffs

Fibre Channel offers a range of scaling options, which trade price for performance. For example, by installing a base configuration of a partially populated switch, ports can be added without disruption while keeping initial costs low. It is important to remember that each additional port on a switch adds--not subtracts--interconnect bandwidth.

Another option is cascading, or connecting multiple switches using one or more switch ports. This configuration is particularly useful in disaster recovery applications where the geographic separation of system elements is important. Switch cascading does affect total system performance since ports have to be dedicated to interconnect switches. And to some extent, throughput performance is affected by heavy traffic through the interconnecting ports.

To maximize performance, the best choice is often a nonblocking switch with a large port count. This type of switch provides maximum throughput and bandwidth in a large, diversified environment. In a cost-sensitive or lower-performance environment, a switch that supports a Fibre Channel loop topology allows the cost of each switch port to be amortized over the many connections on the loop.

Data center fabrics also need to integrate seamlessly with the rest of the enterprise network. Using cascading and the virtually unlimited address space capabilities of Fibre Channel switches (in combination with fabric management tools), administrators can easily connect distributed work groups and departments to the data center. This centralizes backup and allows for high-capacity on-line storage throughout the enterprise, while lowering administration costs and providing economies of scale.

Fibre Channel vs. ATM and Gigabit Ethernet

Fibre Channel. Fibre Channel has roots in networking and highly reliable mainframe environments. As mainframe manufacturers continue to look for ways to drive down costs and as open-systems vendors move to more scalable and reliable environments, Fibre Channel represents the best possible server-storage interconnect technology.

Another benefit of Fibre Channel is its ability to carry TCP/IP or specialized server protocols. This allows users to piggyback server-to-server traffic over the storage network for environments where server-to-server traffic must be isolated from the client/server network. Specialized industries, such as the video and post-production communities, have endorsed Fibre Channel as the connectivity solution of choice and have defined techniques to migrate their high-performance applications to Fibre Channel.

In most cases, Fibre Channel is best for high-speed links to storage systems, server-to-server interconnects, real-time backup, and massive data retrieval. It is not the best technology for connections to the desktop or for wide-area networks (WANs) over extended distances.

Asynchronous Transfer Mode (ATM). Fibre Channel is not the only TCP/IP carrier. Standards define integration of ATM with existing TCP/IP LANs. However, ATM requires a substantial investment in new equipment, tools, and protocol knowledge for forward migration of campus client/server networks.

Telecommunications standards provide mappings for ATM onto WAN carrier technologies as well as inter-switching communication protocols. Support by the telecommunications industry is helping the deployment of ATM in the global enterprise networking market.

In most cases, ATM is best for telecommunications applications, global networks over WANs, and delivery of real-time and multimedia traffic. It is not the best technology for transports that can`t tolerate information loss, storage-area networks (SANs), or simple forward migration of client/server networks.

Gigabit Ethernet. Although Ethernet protocols are used predominately for TCP/IP client/server networks, they also support other protocols such as IPX and DECnet. In many networks today, switched Ethernet networks with Fast Ethernet backbones are connecting switches or are attached as up-links to servers. The natural extension is switched Fast Ethernet networks with Gigabit Ethernet backbones.

Gigabit Ethernet can connect to Ethernet and Fast Ethernet via a simple low-cost bridge. This allows IT departments to protect their investment in drivers. Existing Ethernet systems also provide a minimal learning curve for system administrators. This makes Gigabit Ethernet a popular technology for client/server environments. Gigabit Ethernet adds performance to existing networks and can be integrated with lower-speed Ethernet infrastructures.

In most cases, Gigabit Ethernet is best for client/server networking to support higher bandwidth backbones for Fast Ethernet switches and for aggregating traffic in TCP/IP networks. It is not the best technology for bulk traffic or block-oriented traffic typical of channel/storage attachments, highly reliable backbones that cannot tolerate frame loss, and latency-sensitive traffic.

Each technology has individual strengths and best uses. ATM will remain the technology of choice for WANs, with some penetration in campus and multimedia delivery networks. Gigabit Ethernet will dominate PC LAN backbones and eventually extend to desktops in the high-performance market. Fibre Channel will dominate server/storage backbones and extend into departments and work groups, creating a distributed network for storage applications.

The data center environment includes an increasingly wide variety of server and storage platforms. In these complex environments, it is increasingly important to provide interconnect flexibility and scalability as well as a wide range of price/performance options. The right solution should provide the benefits of the open-systems environment while meeting mainframe-class requirements for enterprise-wide applications and interconnection to existing loop installations. Fibre Channel switching provides these benefits for enterprise storage. Large Fibre Channel switches with high-availability characteristics can be the best fit for these environments because they provide the necessary aggregate bandwidth, balance access contention, and offer predictable latency, high reliability, enterprise-class manageability, and the interconnection flexibility needed to meet IT requirements for mission-critical applications.

Click here to enlarge image

Click here to enlarge image

A 16-port Fibre Channel switch provides an aggregate (flow-through) bandwidth of 1.6GBps.

Click here to enlarge image

How Does Fibre Channel Switching Work?

Fibre Channel`s greatest potential lies in fabric switching. A switch is a multi-port device; each port manages a simple point-to-point connection to attached devices. The interconnect provides concurrent any-to-any connectivity between devices.

From a physical perspective, the topology of a Fibre Channel switch is identical to that of a hub. Both are wired in a star configuration, enabling you to add or remove devices with minimal disruption. However, hubs share bandwidth between devices and have a 1Gbps total (aggregate) bandwidth. A switch port can be attached to a server, peripheral, I/O subsystem, bridge, hub, router, or even another switch. Any port can transmit or receive data at any given time at the full 1Gbps rate. Much like a phone system, the calling device dials the equivalent of a phone number for the destination device and the switch automatically establishes a connection. Multiple calls or data transfers happen concurrently through a multi-port Fibre Channel switch.

In a gigabit-per-second Fibre-Channel-switched network, or multi-point "fabric," storage can be networked and available to multiple host systems, servers, and workstations that use IP, SCSI, or other protocols. Fibre Channel provides highly reliable serial transmission interconnects, replacing bulky cabling and distance-limiting parallel transmission technologies such as SCSI. Fibre Channel also supports an end-to-end bit-error ratio of one error in every 1,012 bits transferred, using 8b/10b encoding, embedded cyclic redundancy check (CRC), and robust bit-error checking algorithms.

Peter Dougherty is director of Fibre Channel product marketing at McData Corp., in Broomfield, CO. He is also chairman of the Fibre Channel Association`s Interoperability Working Group.

This article was originally published on April 01, 1998