Extending storage over WANs

Choosing the best solution first requires an understanding of latency, data integrity, and bandwidth utilization.

By Gary Johnson

Business continuity and disaster recovery have taken on a very high profile, requiring more and more organizations to extend their storage over long distances using a wide area network (WAN). Not only can losing access to critical data for even a short period of time cause enormous costs, but companies are now faced with an increasing array of laws and regulations that dictate recovery time.

Just a few months ago, three US regulatory agencies issued disaster-recovery requirements that mandated that large banks and other financial services firms be able to recover their operations within the same business day a disaster or disruption occurs.

Defined as a distance of 100 miles or more, the extension of storage over a WAN brings a unique set of considerations to most organizations, especially if it's their first time—because these are requirements they would not have previously dealt with in either LANs or MANs. Three key issues are latency, data integrity, and bandwidth utilization.

In our first article (see "Extending storage over WANs, part I," InfoStor, July 2003, p. 38), we provided an introduction to the topic of storage over a WAN. In part III, we'll tell you how you can deal with the issues to help you better develop a plan that will effectively extend your organization's storage network over long distances. The last part in the series will highlight the various technologies in use today.

Issue #1: Latency

The first issue in long-range storage is directly related to distance. Even traveling at the speed of light, the delay of data transfer between sender and receiver increases as distance increases. This phenomenon is known as latency.

Storage network designers commonly use the following guideline for predicting latency: 1 millisecond (ms) for each 100 miles. This guideline is used regardless of the amount of WAN bandwidth and helps account for any difference in the distance between locations "as the bird flies," versus what the actual cable path distance might be.

If, for example, the distance between the server and a tape drive is 500 miles, then the one-way transfer latency is 5ms. However, when a server is writing a block of data to tape, the I/O is not complete until the tape acknowledges receipt of the data by sending a response back to the server. Thus, the true latency of a data transfer is really the round-trip time for completion of an I/O. In the case of 500 miles, the actual latency to complete one command is 10ms plus the queuing time of the network equipment at both ends (see figure).

Latency impact on disk

Extension of primary disk storage is rare, due to its critical performance and availability criteria. What we are referring to here is latency relative to a second-copy disk that is extended over distance. Second copies can be performed either synchronously or asynchronously. Synchronous copying means the I/O is not complete until the second copy is written and a "good status" response is returned. With asynchronous copying, multiple I/Os can be "in-flight" at one time.

For a transaction-processing application, 10ms is an eternity when reading or writing to a primary disk. These types of applications are highly tuned to optimize overall I/O performance. Multiple paths to the disk subsystem are created with multiple volumes (physical and logical) defined to increase the number of concurrent I/Os to the database. When you consider that some transaction processing applications handle hundreds or thousands of read/write requests per second, the 10ms delay backs up subsequent transactions to the point that response time is measured in seconds or worse.

This is unacceptable to the application and, subsequently, to end users as well. This is analogous to a multi-lane freeway where the lead car in each lane stops for a period of time and then continues moving. Just one car stopping can cause a traffic backup. The length of that backup can increase with the number of stops and/or the length of the stop.

Latency impact on tape

Tape backups and restores are offline activities that are not coupled to an end-user application. The only time tape activity will impact an end-user application is if it steals server cycles while the application is also active. While latency on disk I/O affects user applications, latency on tape I/Os impacts time to backup and time to recover data. Latency of 10ms will cause the tape drive to start and stop with each I/O command, significantly degrading its performance. In fact, with tape, data transfer performance will begin to degrade at 50 miles. At 100 miles, performance is reduced to 40% to 50% of native throughput.

The latency of a data transfer over a WAN is 10 milliseconds plus the latency of the WAN equipment (e.g., storage routers) on both ends.
Click here to enlarge image


To solve this issue, a technology called "pipelining" has been developed, which we'll discuss in the next article in this series.

Issue #2: Data integrity

Data integrity refers to keeping the contents of a data block from changing as it is moved from the sender to the receiver. Data corruption occurs when the value represented by the data byte is changed to some other value. When this happens, the integrity of the data is lost. The more electronic components a byte of data must pass through between the server's memory and a storage device, the higher the probability that the data byte will be corrupted.

In a local environment, servers and storage devices use parity, checksums, etc., to ensure the integrity of data. When you extend storage over a WAN, additional components such as routers, gateways, and multiplexers handle the data blocks. With gateways and some routers, the data blocks are converted from a channel protocol format, such as ESCON or Fibre Channel frames, to a networking protocol such as HDLC or IP. The data blocks typically pass through the mid-plane and memory of these WAN devices, which increases the risk of data corruption. Just passing through a physical link can subject the data blocks to outside interference and cause corruption. Many of these devices have some degree of protection by using parity bits or checksums; however, with these types of checks there is still a statistical probability that corruption may occur.

Solving the issue of data integrity involves a feature called end-to-end cyclical redundancy check (CRC), which enables reliable data copying, or "mirroring." We'll discuss this feature in more detail in the next article.

Issue #3: Bandwidth utilization

Bandwidth is the most expensive component of any storage over MAN/WAN solution in terms of total cost of ownership (TCO). While the equipment required is a one-time, up-front cost, the recurring costs are significant. For example, a recent bid for two OC-3 circuits from Minneapolis to Chicago cost about $33,000 per month—or almost $400,000 per year.

The MAN/WAN connectivity between servers and storage devices is sometimes referred to as the "pipeline." The issue here is distance and balancing the use of multiple links. If the server waits for each I/O to return with completion status before sending another data block (synchronous operation), at 500 miles the bandwidth usage (utilization) on one circuit will be less than 25%. To better use your bandwidth, many I/Os need to be in the pipeline. In other words, the server needs to be streaming I/Os to the storage device.

The actual number depends on the distance and size of the data blocks. Although storage buffering techniques are useful for optimizing local performance, these techniques over a WAN create data transfer delays and significantly degrade bandwidth utilization.

For tape, you can run multiple streams to improve bandwidth utilization; however, each stream suffers from the associated 10ms of latency, which significantly degrades tape performance.

If a second-copy disk also runs synchronously to the application, the 10ms delay has the same effect on bandwidth utilization. Again, I/Os to multiple volumes will increase bandwidth utilization, but application response time on each I/O will still be impacted by the 10ms latency. If the second-copy disk is operating asynchronously with the application or the primary disk, then bandwidth utilization can be improved by allowing several secondary I/Os to be outstanding at one time.

Some second-copy applications are managed in the server and others are managed in the primary disk controller. The first concern is data consistency between the primary and secondary disks. The higher the number of outstanding I/Os, the less data consistency you will have.

The second concern is the amount of memory that will be required in the server or the amount of cache that is required in the storage controller to store the I/Os until a completion response is received from the second-copy disk.

Multiple links are used for availability in terms of a link failure. If one fails, the server will still have available bandwidth to its storage. If you are paying for multiple links, you certainly want to use this available bandwidth to ensure each link is fully utilized. To accomplish this, you need to negate the effects of latency and balance the I/Os equally across all links. The storage data transfer protocol cannot accomplish this without assistance.

Optimizing available bandwidth is a fundamental responsibility of the storage router. To meet this objective and solve the bandwidth utilization issue, four technologies come into play: pipelining, payload matching, load leveling, and compression, which will be covered in the next article.

Gary Johnson is vice president of solutions at CNT (www.cnt.com) in Minneapolis, MN.

This article was originally published on August 01, 2003