Serial ATA, Serial Attached SCSI, and/or Fibre Channel? Here are some guidelines to help you make the call.
By Arun Taneja
Serial ATA (SATA) disk drives are taking the enterprise by storm, winding up in places for which SATA was not originally intended. Simultaneously, several vendors are enhancing SATA to make it robust enough for enterprise-level requirements, most notably in the areas of performance, reliability, and availability.
At the same time, Serial Attached SCSI (SAS) disk drives are just beginning to hit the market in servers and external subsystems. All of this activity leads to the question of whether end users should implement SATA, SAS, or Fibre Channel (FC) at the disk drive level and/or host connectivity level.
This article addresses a variety of technologies that are required to allow SATA, SAS, and Fibre Channel to most effectively play the roles they were designed for. We also look at the interfaces’ pros and cons to help users decide for themselves what the right solution is for their requirements.
SATA in the enterprise
The drivers for SATA playing a big role in the enterprise include users’ need to use secondary disk for backup and recovery; the increased understanding among users that active archiving is distinct from backup and restore and is most effectively done using secondary disk; and the enterprise’s ongoing requirement for 24x7 operations and increased pressures on reducing recovery time objectives (RTOs) and recovery point objectives (RPOs).
In addition, small to medium-sized businesses (SMBs) are experiencing even faster capacity growth than large enterprises and are discovering that their storage and data-protection requirements have more similarities than differences relative to large enterprises.
All of these trends are leading to the use of higher-capacity, lower-cost disks, exemplified by SATA. These drives exhibit lower rotational speeds and are designed for lower duty cycles than Fibre Channel or SAS drives, but that’s adequate for many of the applications mentioned above. As a result, SATA has taken off like a rocket over the past year and will continue its dramatic climb for at least the next three years.
(This article focuses on SATA in non-desktop/ laptop environments. SATA’s rapid incorporation into desktops and laptops is driving the economies of scale that make SATA attractive in the enterprise.)
The relative value of SATA over Fibre Channel and even SAS drives has been so strong that many users have deployed SATA drives as primary storage. A good example is Network Appliance’s SATA-based NearStore, which was positioned by the company as a backup, restore, and archiving platform but is often deployed as primary storage. Enterprises are finding that SATA is “good enough” for many applications that are not considered mission-critical. While this goes counter to some disk drive manufacturers’ desire to keep SATA in a lower strata versus Fibre Channel or SAS (so those vendors can continue to charge a premium for FC and SAS drives), users ultimately decide what they will deploy and where. In fact, due to the favorable economics, the pressure on IT is so high to use SATA that it is being used in many applications for which it was not originally intended. This will increase due to “adjunct” technologies that narrow the gap between SATA and FC/SAS.
SAS in the enterprise
SAS-based storage systems (or storage systems based on a SAS infrastructure, which also supports SATA drives) are just beginning to show up in the enterprise. The most logical way to look at their role is as a replacement for parallel SCSI. A large percentage of SAS drives will show up in direct-attached storage (DAS) configurations, with SAS adapters or expanders directly on the server motherboard, especially in mid-tier products. But SAS will also show up in external storage in the form of JBOD arrays, where SCSI usage was declining in favor of Fibre Channel. There are several reasons for this, the most important being SAS’s ability to natively incorporate SATA drives using the SATA Tunneling Protocol (STP) in the same physical enclosure as SAS drives, with 100% compatibility.
Once we have defined more details on SAS, SATA, and Fibre Channel from a technology and trends perspective, it will become clear how strong a role SAS will play in the future.
FC in the enterprise
Fibre Channel is solidly established in the enterprise for mid-tier and high-end primary storage applications. Fibre Channel provides high availability, performance, scalability, and high duty cycles. Almost all high-end disk arrays use Fibre Channel at the disk drive and host connectivity levels. Despite competition from SAS and SATA, Fibre Channel will continue to thrive because of continued enhancements and price reductions in both drives and other components such as host bus adapters (HBAs) and switches. It is very difficult to displace an incumbent technology if it performs well and continues to decline in cost.
The key question is how these three technologies will balance out over time.
SATA is being rapidly enhanced by a number of vendors. Often, these vendors have no legacy SCSI or Fibre Channel business that might be jeopardized by SATA. (One exception is Emulex, which has made some strategic moves to incorporate SATA into the Fibre Channel fold.) To understand these enhancements it’s necessary to understand the limitations of SATA drives in enterprise storage environments.
The limitations that are most profound in SATA drives include lack of a dual-port capability and lack of diagnostics regarding drive operation. A number of vendors are also enhancing the way in which SATA drives interface with SAS and Fibre Channel infrastructures. Examples of SATA enhancements include:
Converting SATA to dual port
Dual-port capability at the drive level is required to meet minimum availability requirements in the enterprise, even for nearline applications. Fibre Channel and SAS drives are both natively dual-ported, providing two independent paths from the server to the disk drive. If one path is lost, data is read or written via the other path. The most common method used to provide two independent paths into a SATA drive is an “interposer” card that is installed inside the drive canister itself.
There are two basic approaches. One, exemplified by Marvell and Vitesse, is the use of either an active-passive (A/P) or pseudo active-active (pA/A) multiplexer on the interposer card. An A/P mux works as implied: The active path is the way to the drive when all is well, while the passive path comes into play only if the active path fails. Think of it as half duplex, or one path at a time. A pA/A mux improves on the A/P mux by keeping both ports alive, but it requires a switching mechanism and involvement of upper-level fail-over software in order to switch over to the alternate path.
The second approach is exemplified by SiliconStor, which ships a product called AAMUX that converts a SATA drive to an active-active dual-ported drive similar to Fibre Channel or SAS drives. The key advantage of an A/A approach is reduced latency when a failure occurs and reduced software complexity. In an A/P approach, a heartbeat link is used to tell the upper-level hardware and software that a path has failed. During a fail-over operation, specific procedures have to be invoked, adding software complexity, switching latency, and possibly cost. In the case of the A/A approach, the fail-over is transparent and instantaneous at the drive level, as it is with Fibre Channel or SAS drives. The A/A approach also enables load balancing across two independent data paths.
Since the interposer card is implemented at the disk drive level, it becomes an excellent place to add functionality and diagnostics. Each vendor has its own list of features. SiliconStor, for instance, sniffs out a number of variables from the drives and draws predictive conclusions regarding failures. SiliconStor’s AAMUX also can be programmed to start up the drive periodically and check its health.
Another variation of the A/A approach is taken by Sierra Logic, which has an interposer card that includes an FC-to-SATA bridge that does Fibre Channel-to-SATA protocol conversion at each drive. Sierra Logic’s approach yields a SATA drive with a dual-ported Fibre Channel interface. The drive can be transparently attached to an FC-AL loop just like any other Fibre Channel drive. However, since the bridge circuits are more complex, they make the interposer card somewhat larger and more expensive relative to the other approaches, creating a different value proposition.
We expect additional entries in the A/A mux category from a number of other vendors soon.
SATA tunneling over FC
Another technology relevant to SATA is the FC-SATA tunneling protocol proposed by Emulex and now being considered for standardization by the ANSI T11 committee. Essentially, Emulex’s FC-SATA proposal is a new protocol extension that uses the Fibre Channel transport. It works for both point-to-point and FC-AL environments. Fibre Channel HBAs and I/O controllers need to be modified via firmware in order to recognize and deal with this protocol extension. Emulex is also adding support for this in the next generation of its InSpeed embedded Fibre Channel switches. Some component vendors are expected to follow suit.
The key benefit of this FC-SATA specification is that SATA drives can be accommodated in a Fibre Channel infrastructure seamlessly and with minimum latency. Fibre Channel and SATA drives could even be mixed in the same tray. In contrast, if you want to support SATA drives in a Fibre Channel environment today you need to bridge from SATA to Fibre Channel either at the tray or drive level, as in the case of Sierra Logic’s FC-SATA bridge. For example, an FC-FC RAID controller today can support Fibre Channel drives in one tray and SATA drives in a second tray, with bridging at the tray or drive level. Bridging is a well-tested way to convert from one protocol to another, but it comes at a price. A bridge can add $600 to $900 at the tray level. For SATA, this can be a large penalty.
Note that the standardization of this proposal still has an uphill battle, because some key vendors (such as Emulex competitor QLogic) have not yet signed on to support this standard. The value proposition, however, is strong enough that the initiative is expected to be ratified.
Seagate has developed a low-cost drive with a Fibre Channel interface (the NL35 series) integrated on the drive controller, which is conceptually similar to using an FC-SATA bridge interposer card, but with a much higher degree of integration. These drives are sometimes referred to as FATA drives since they combine elements of Fibre Channel (the interface) and SATA (low cost).
In addition, some vendors support multiple-speed (rotational rate and interface speed) Fibre Channel drives within the same chassis and present this as tiers of storage. EMC, for example, supports storage tiers in this manner in its DMX disk arrays. This is a way to counter the need for SATA while staying with a pure Fibre Channel environment. However, the cost differential between Fibre Channel and SATA drives is so large that this approach will likely only work with large, expensive, monolithic arrays.
SAS was developed by the SCSI Trade Association (www.sta.org) to overcome some of the deficiencies of SATA and is in a sense a superset of SATA. (SAS was also designed as a defensive measure to ensure that SCSI would not lose to Fibre Channel and SATA.)
The drawbacks to parallel SCSI are well known. Its bus-based architecture only allows a small number of drives per controller; cabling is a disaster in terms of length and size; and at 320MBps the spec was maxed out. Going to higher speeds would have required a compromise on cable lengths, which were already too short for today’s requirements. The most brilliant move made by STA was to make the physical spec for SATA and SAS fully compatible, which allows OEMs, integrators, and end users to mix SAS and SATA drives within the same chassis.
SAS advanced SATA in the dimensions of performance, reliability, and expandability. Since SATA was originally designed for the desktop, it wasn’t scalable. If you wanted to add another drive, you had to add another SATA controller, too. The SATA II specification defined a hub that allows for attachment of up to 15 drives, but this requires that SATA 1.0 controllers be replaced. Also, the hubs could not interconnect with each other, requiring additional controllers for large-scale SATA configurations.
In addition, SATA performance was limited by the fact that only one hub port could be the initiator, and the initiator could only talk to one drive at a time, with a requirement for a reset in between. The resets caused all data from the previous communication with the host to be lost. There was no concept of dual paths to the host, and the cabling from the hub to each drive could only be one meter. In high-performance enterprise environments, these limitations excluded SATA.
SAS solved all these deficiencies of SATA, while maintaining 100% compatibility at the physical level. SAS drives are designed for 24x7 full-duplex operation and support the bulletproof SCSI protocol. SAS has complete fail-over capability and FC-like expandability. High-speed SAS drives that rival Fibre Channel drives in performance and reliability are just hitting the market and will show up in storage systems over the next few months.
Expandability in SAS is defined differently than with other drive interfaces. SAS defines two types of expanders-edge and fan-out, both of which are switches similar in concept to Fibre Channel switches. An edge expander can connect up to 128 disk drives, either SAS or SATA. A fan-out expander can connect up to 128 edge connectors, effectively creating a single SAS domain as large as 16,384 drives. While not a practical configuration, it is theoretically possible to design a 16,384-drive SAS/SATA system out of a single RAID controller with one SAS port. As with Fibre Channel, multiple SAS ports (initial interface speed is 300MBps) can be trunked together, with built-in load balancing and fail-over protection. SAS expanders are just coming into the market. For instance, PMC Sierra, Vitesse, and LSI Logic are shipping expanders for use in enterprise environments.
With some exceptions, SAS rivals Fibre Channel in terms of performance (300MBps versus 400MBps), reliability (dual-port drives and active fail-over), and data integrity (unique addressability at the drive level). SAS’s 8m cabling is adequate for some data-center configurations, although it doesn’t match Fibre Channel’s 10km fabric connectivity distance.
Enhancements applied externally to SATA drives to enable their use in the enterprise apply to SAS infrastructures as well. SiliconStor, for instance, has upgraded its interposer card to provide dual-ported SATA drives with the right signaling levels for use in a SAS environment. The SiliconStor interposer also conditions the SATA signal and re-times it to be consistent with enterprise-level systems.
Obviously, several technologies are vying for a position in the enterprise. Migrations are always painful, so deploying the right technologies at the right time is critical to maximizing IT efficiency. How should end users make a decision, given the state of the market today?
In our opinion . . .
Only 18 months ago we thought that the new storage world would predominately consist of Fibre Channel and SATA-Fibre Channel for both disk drives and fabric connectivity and SATA for disk drives-with SCSI fading away. For high performance, reliability, scalability, and availability, FC-FC (drive-host) configurations would dominate. For secondary (Tier 2) storage, FC-SATA would dominate.
But SCSI in the form of SAS is coming back with a vengeance. If it had not been for the brilliance of supporting SATA so seamlessly, SAS would have been too little too late. But the ability to create two tiers of storage within the same domain is very attractive. As a reaction to this seamless SAS-SATA interaction, Emulex defined the FC-SATA protocol extension, which could allow integrators to smoothly integrate SATA into Fibre Channel infrastructures.
SAS has more or less equaled Fibre Channel and is here to stay. Server vendors will put SAS adapters on their motherboards, especially for the high-volume midrange and below markets, just like they did with SCSI adapters. This would encourage the use of both SAS and SATA drives.
For very high-end servers, we expect the I/O interface of choice to remain Fibre Channel. At the midrange and high-end of the market, Fibre Channel and SAS will duke it out. For high-end external storage systems (e.g., EMC DMX, IBM DS8000, HDS TagmaStore), we expect FC-FC to continue to dominate. But we expect midrange storage systems to be SAS-SAS or even FC-SAS (Ario Data recently announced an FC-SAS RAID controller), although SATA drives will also be mixed into those configurations. For example, cost-conscious users may opt for SAS-SATA or FC-SATA configurations.
In terms of SATA drives in enterprise applications-whether it’s for backup/restore, archiving, or Tier 2 or Tier 3 primary storage-users should consider “adjunct” technologies that bring SATA to a minimal level of acceptability in the enterprise. We favor an active-active mux that provides not only dual ports to SATA drives but also increased reliability and diagnostics features. When used in conjunction with an FC-SATA router, A/A muxes also deliver increased throughput. And when used in conjunction with an HBA that has been updated with the proposed FC-SATA tunneling protocol, A/A muxes will still deliver the same benefits.
Over time, we expect many of these adjunct features to be integrated into SATA drives, despite some drive manufacturers’ desire to keep SATA “down” in the perceived hierarchy. The economics and capacity of SATA drives are simply too appealing to be suppressed by certain drive manufacturers. The innovation of chip-level players such as Marvell, PMC Sierra, Sierra Logic, SiliconStor, Vitesse, and others will eventually force drive manufacturers to integrate new technologies into SATA drives for the enterprise. But that could take years. In the meantime, cost-effective interposer approaches will have to shoulder the burden of making SATA drives full members of the enterprise IT landscape.
Just when you thought storage decisions were going to get simpler, in comes SAS to crash the party. The key now is to understand the relative advantages and disadvantages of each technology and choose what fits your specific requirements.
Arun Taneja is a consulting analyst and founder of the Taneja Group (www.tanejagroup.com) in Hopkinton, MA.