Part II - Building storage networks: A book excerpt

Storage networking is built on three fundamental components: wiring, storing, and filing.


Storage networking provides storage applications on any number of suitable wiring technologies. In general, storage networking products have been associated with specific network technologies. Storage area network (SANs) have been associated with Fibre Channel technology, and network-attached storage (NAS) is considered to be an Ethernet technology. Unfortunately, identifying storage network technologies with specific data networks has not helped people understand the abstract architectural components of storage networking. (See InfoStor, September 2001, p. 28, for Part I of the book excerpt.)

Storing and filing as network applications

Filing is familiar as a client/server application where both client and server perform similar communications functions. For example, a server for one group of clients may itself be a client of some other server. It is strange to think about it as such, but on a communications level, not an application level, clients and servers are peers.

Storing, however, is built on a different type of relationship. Storing-level communications is based on a master/slave model where host system initiators are master entities issuing commands and storage devices/ subsystems are slave entities responding to those commands. In general, the slave entities have much less flexibility than the masters that direct their operations. Notable exceptions to this arrangement include devices and subsystems with implemented embedded initiator functionality such as disk drives with integrated XOR processing and backup equipment with third-party-copy capabilities. Even in these cases, however, the embedded initiator in the device is used for specific applications and not for general-purpose storage communications.

Figure 1: The hierarchy of storing and filing in a single system.
Click here to enlarge image

There is an implied hierarchy between storing and filing, where users and applications access data on the filing level and where filing entities such as file systems and databases access the data on a storing level. This hierarchy exists as an internal relationship within nearly all systems used today. This hierarchy, along with the corresponding I/O stack functions, is depicted in Figure 1.

Although a hierarchy exists between storing and filing, it is not always necessary for it to be implemented as in Figure 1. Filing can access the wiring function independently without first passing through a storing function, as shown below.

Click here to enlarge image

The preceding drawing is the scenario usually used to show how NAS systems work. Analyzing the I/O path in more detail, however, one realizes the necessity for the client/server filing operation to be converted to a master/slave storing function and transmitted by the server over some sort of wiring to the destination storage devices. This conversion is done by a data structure function within the server's file system that determines where data is stored in the logical block address space of its devices or subsystems.

For most NAS products today, the wiring function used for storing operations is a storage bus. When all the pieces of the I/O path are put together for NAS, we see that the NAS system provides filing services to network clients and incorporates some type of storing function, typically on independent sets of wiring, as shown above.

Click here to enlarge image

While individual NAS vendors and their particular products may have specific storing and wiring implementations, no architectural requirements for storing or wiring are implied by the NAS concept. Therefore, NAS is considered to be mostly a filing application that uses the services provided by storing and wiring. Although a particular NAS product may implement specific wiring and storage technology, the primary external function provided to customers is its filing capabilities.

SAN as a storing application
Storing functionality can be generalized as the master/slave interaction between initiators and devices. Storing is deterministic by design to ensure a high degree of accuracy and reliability. To some degree this is a function of the underlying wiring, but it is also a function of the command sequences and exchanges used in storing. Several storing technologies are available, the most common being the various flavors of SCSI commands.

It can be very hard to separate the storing function from the wiring function when one looks for product examples. For instance, a Fibre Channel host bus adapter (HBA) is certainly a part of the wiring in a storage network, but it also provides functionality for processing SCSI-3 serial data frames. It is important to realize that the SCSI-3 protocol was developed independently of Fibre Channel technology and that nothing inherent in SCSI-3 ties it to Fibre Channel. It is independent of the wiring function and could be implemented on Ethernet or many other types of network.

Similarly, there is no reason another serial SCSI implementation could not be developed and used with Fibre Channel or any other networking technology. In fact, there is no reason that SCSI has to be part of the equation at all. It is one of the easiest storing technologies to adopt because it has been defined for serial transmission, but there certainly are other ways to control devices and subsystems.

So what is a SAN? It is the application of storing functionality over a network. SANs by definition exclude bus types of wiring. SANs provide deterministic control of storage transmissions, according to the implementation details of the storing protocol used and the capabilities of the underlying network.

Aligning the building blocks of storage networking

Storage networking is certainly not "child's play," but that doesn't mean we can't approach it that way. Certainly the SAN industry has made a number of ridiculous puns and word games surrounding SAN and sand, so with that as an excuse, we'll discuss building blocks. The three building blocks we are interested in, of course, are these:

Click here to enlarge image

As discussed previously, the implied and traditional hierarchy of these building blocks within a single system is to place wiring on the bottom and filing on top, such that storing gets to be the monkey in the middle, like this:

Click here to enlarge image

Of course, in the worlds of NAS and SAN, these blocks have been assembled like this:

Click here to enlarge image

But if we want to take a detailed view of NAS, we know that NAS actually has a storing component as well, which is often parallel SCSI, and we place the building blocks within client and server respectively, like this:

Click here to enlarge image

But as we've been saying in this article, wiring is independent from both storing and filing and, in fact, can be the same for both. So we've structured the building blocks of filing (NAS) and storing (SAN) on top of a common wiring, like this:

Click here to enlarge image

Now the preceding drawing is probably only interesting in theory, as something to illustrate the concept. In actual implementations, it is probably a good idea to segregate client/server traffic from storage traffic. This provides the capability to optimize the characteristics of each network for particular types of traffic, costs, growth, and management.

That said, it might also be a good idea to base the two different networks on the same fundamental wiring technology. This allows organizations to work with a single set of vendors and technologies. As long as a common wiring technology can actually work for both types of networks, there is the potential to save a great deal of money in the cost of equipment, implementation, training, and management. This type of environment, shown in Figure 2, includes a storage device as the final destination on the I/O path.

Race for wiring supremacy

Three networking technologies have the potential to provide a common wiring infrastructure for storage networks. The first is Fibre Channel, the next is Ethernet, particularly Gigabit Ethernet, and the third is InfiniBand. We'll make a brief comparison of their potential as a common wiring for storage networks.

Figure 2: Common wiring, but separate networks for filing and storing.
Click here to enlarge image

Fibre Channel strength
Fibre Channel's primary strengths are precisely where Ethernet has weaknesses. It is a high-speed, low-latency network with advanced flow control technology to handle bursty traffic such as storage I/O. However, its weaknesses are the major strengths of Ethernet. The Fibre Channel industry is still small compared to Ethernet, with limited technology choices and a relatively tiny talent pool for implementing and managing installations. The talent pool in Fibre Channel is heavily concentrated in storage development companies that have a vested interest in protecting their investment in Fibre Channel technology. This does not mean that these companies will not develop alternative wiring products, but it does mean that they will not be likely to abandon their Fibre Channel products.

Of the three technologies discussed here, Fibre Channel was the first to develop legitimate technology for common wiring. But technology alone does not always succeed, as has been proven many times throughout our history. The Fibre Channel industry has never appeared interested in its potential as a common wiring. Although it has a technology lead, having begun as the de facto standard for SANs, it is extremely unlikely that Fibre Channel will cross over to address the NAS, client/server market.

Ethernet's move
Ethernet has the obvious advantage of being the most widely deployed networking technology in the world. There is an enormous amount of talent and technology available to aid the implementation and management of Ethernet networks. While the 10Mbps and 100Mbps Ethernet varieties are sufficient for NAS, they are probably not realistic choices to support SANs because of their overall throughput limitations and lack of flow control implementations. Therefore, Gigabit Ethernet would likely be the ground floor for storing applications such as SANs. However, even though Gigabit Ethernet has the raw bandwidth and flow control needed for storage I/O, most Gigabit Ethernet switches do not have low enough latency to support high-volume transaction processing.

There is little question that Ethernet will be available to use as a common wiring for both filing and storing applications, but its relevance as an industrial-strength network for storing applications has to be proved before it will be deployed broadly as an enterprise common wiring infrastructure.

InfiniBand in the wings
The latest entrant in the field is InfiniBand, the serial bus replacement for the PCI host I/O bus. InfiniBand's development has been spearheaded by Intel with additional contributions and compromises from Compaq, Hewlett-Packard, IBM, Sun, and others. As a major systems component expected to be implemented in both PC and Unix platforms, InfiniBand is likely to become rapidly deployed on a large scale. In addition, a fairly large industry is developing the equivalent of HBAs and network interface cards for InfiniBand. Therefore, InfiniBand is likely to grow a sizable talent pool rapidly.

In relation to storage networks, the question is: Will storing and/or filing applications run directly across InfiniBand wiring, as opposed to requiring some sort of InfiniBand adapter? Immediately, soon, years away, or never? The technology probably needs to gain an installed base as a host I/O bus before it can effectively pursue new markets such as storage networking. However, InfiniBand certainly has the potential to become a legitimate storage wiring option at some point in the future.

Click here to enlarge image

As the apparent method of choice for connecting systems together in clusters, along with their associated storage subsystems, this could happen sooner than expected. As with any other networking technology, it is not so much a question of whether the technology can be applied but rather when attempts will be made and by whom with what resources.

Common wiring
There aren't any crystal balls to predict the future of storage networking. However, any time functions can be integrated together in a way that reduces cost and complexity, the only question is whether it can be marketed successfully. Common wiring is more than a theoretical abstraction for storage networks, but it represents a large opportunity to integrate data networks and storage channels under a single technology umbrella.

As Fibre Channel, Ethernet, and InfiniBand technologies evolve in response to this integration gravity, it is almost inevitable that NAS and SAN developers will look for ways to combine functionality, and their products will look more and more alike. The terms NAS and SAN will seem completely arbitrary or obsolete, and it will be necessary to distinguish storage products by the storing and filing applications they provide, as opposed to the limitations of their initial implementations. At that point, a whole new level of storing/filing integration will become visible and true self-managing storage networks may be possible. But first, the wiring slugfest!

The table briefly summarizes the competing technologies that could be used to form a common wiring and their current status.


This article discusses the fundamental components of storage networks-wiring, storing, and filing-in relation to the most common applications of storage networks today: NAS and SAN. More than just the similarity of the acronyms used, NAS and SAN have confused the industry and the market because of their similarities and the lack of an architectural framework to view them in.

NAS, the application of filing over a network, has two important roles. First, it provides a service that allows applications and users to locate data as objects over a network. Second, it provides the data structure to store that data on storage devices or subsystems that it manages.

SAN, on the other hand, is the application of storing functions over a network. In general, this applies to operations regarding logical block addresses, but it could potentially involve other ways of identifying and addressing stored data.

Wiring for storage networks has to be extremely fast and reliable. Fibre Channel is the incumbent to date, but Gigabit Ethernet and InfiniBand are expected to make runs at the storage network market in years to come. The development of a common wiring infrastructure for both filing (NAS) and storing (SAN) applications appears to be inevitable, and it will deliver technology and products that can be highly leveraged throughout an organization.

Marc Farley is a storage professional and author of Building Storage Networks, First and Second Editions.

This article is excerpted with permission from Building Storage Networks, Second Edition, by Marc Farley (Osborne/ McGraw-Hill, ISBN 0-07-213072-5, copyright 2001).

This article was originally published on October 01, 2001