Henry Newman's Storage Blog Archives for October 2011

PCIe Roadmap Needs to Find a More Popular Route

I have been thinking about the PCIe roadmap where it was announced that performance will double by 2015 or maybe 2016.

At the same time, CPU performance is going to up by more than 2x, CPU communications channels like HT and QPI have more than doubled, memory performance has gone up more than 2x and SAS performance has likely more than doubled. Here is the SAS Roadmap. Note that 3.0 Gbit overlapped PCIe 1.0, and PCIe 2.0 and 6 Gbit will overlap PCIe 2.0 and 3.0, supposedly soon to arrive. We will get 12 Gbit in 2013.

6 Gbit=768 MB/sec is about 1.5 PCIe 2.0 lanes per SAS lane. 12 Gbit is of course also about 1.5 PCIe 3.0 lanes. The SAS roadmap and the PCIe roadmaps and releases do not see to match very well. PCIe lanes are usually 4 and 8 now sometimes 16. Clearly SAS lanes and PCIe lanes are a major mismatch. (e.g., 4 SAS 6 Gbit lanes equal 6 PCIe lanes). Besides the mismatch on performance with SAS and PCIe, I think performance is lagging, but that is another topic. We will get 24 Gbit SAS and be stuck on PCIe 4 at 2 GB/sec per lane and still have the mismatch with 3 GB/sec per SAS lane. This mismatch, given the number of PCIe lanes most vendors provide and the number of SAS lanes most vendors provide, means that you either waste PCIe lanes to ensure the SAS card runs at rate or waste SAS lanes if you oversubscribe the PCIe bus. This does not make for good balance, and maybe the PCI-SIG should think about matching the SAS performance rather than going down a separate patch.

Many believe that long-term SAS will be the dominant storage connectivity market space. It is about time the PCI-SIG get in lock step with one of its biggest users.

Labels: standards, SAS, PCIe

posted by: Henry Newman

Some Final Thoughts on Data Integrity

This is a call to the industry to make changes to standards and to assume that files are not going to be 100 percent reliable 100 percent of the time. I think it is time to get the bodies that control the standards for various file formats to provide frameworks for easier file recovery.

I have said this before, but I think it deserves repeating again. We need to have a per-file method to address data corruption. Standards that maintain file format (such as tiff, jpg and other media formats) and vendors (such as Adobe, which controls PDF) need to think about things such as multiple file headers, checksums and error correction codes (ECC).

When a file is created, the creator should determine the level of error correction as only he knows how important the file is. The importance of a file could, of course, change over time. At that point, the file should be re-read and re-written based on the new level of importance, and hopefully the file will be intact. We cannot make the expectation that the hardware industry ensure all files are completely protected, as the cost is far too high. The consumer is driving the storage market, and consumer markets do not pay for anything they cannot cost justify; high data integrity might float my boat, but it will not drive the market.

Many applications when you are editing a jpg file allow you when you save it to specify the compression to a small, medium or large size for example. I believe that error correction should have a similar interface. Small might be just a checksum to tell you something has changed, while large might write enough ECC so that the file is essentially mirrored. So far, since I wrote about this about a year ago nothing has happened, and no one is speaking about it as least as far as I know.


Labels: high availability, standards, data integrity, Storage

posted by: Henry Newman

More Thoughts on Data Integrity

Let's assume you've accepted the fact that you are not going to get 100 percent of your data back 100 percent of the time. What should your expectations be?

My view is that someone should calculate the expected data loss and provide some kind of guarantee. The question I have been thinking about is how that can apply to someone with less than the data loss volume. Let's say a vendor provides a 10/9 guarantee against data loss. Another way to look at it is 112,590 of bytes lost per petabyte (1024*1024*1024*1024*1024 bytes). If I have only 40 TB of data in the archive, should I expect 40/1024*112590 bytes of data loss (4,398)?

This is totally unrealistic. What is the 112,590 bytes? Was a single byte in 112,590 PDF or JPG headers? That could mean 112,590*8MB (average size of the pdf or jpg) or, even worse, headers from mpgs. We are now talking about over 879 GB. Clearly, in my 40 TB archive that would be totally unacceptable. But what is acceptable, and how could this become auditable? That is the big question. First of all, the audit must be independent, and it must be based on a documented set of standards that need to be agreed to similarly to ISO standard. The problem with ISO audits is that many are based on internal people in the company doing the audit. That will not work in the case of data loss, and too much will be at stake.

I do not see this situation easily resolved, but there are many examples in other markets and industries, such as external account audits when something bad happens, that could be used as example. I am not sure that our industry is ready for the complex regulation that exists in accounting, but by the same token I am not sure that archives, archive vendors and the people that use them do not need the protection.

Labels: high availability, disaster recovery, data loss, data integrity, recovery plan, five nines

posted by: Henry Newman

Resetting Expectations for Data Integrity

We all think about availability in 9s.

For data, we all think about 100 percent data integrity, which is totally unrealistic. We should look at data integrity in terms of 9s also. Think about this:

Clearly, with large amounts of archival data you are still going to have data loss even at 20 9s. How do you even calculate the potential of data loss with all of the hardware and software in the datapath? I believe it is time that the discussion change from our current thinking that you are not going to lose any data to the realization that data loss is going to happen and how are we going to address it and deal with it? The problem is you do not know what you are going to lose, as there is no emphasis or standards that allow you to define the level of importance of the data. There are, of course, proprietary frameworks that allow you to define the importance and potentially save more copies of the file, but you are locked into a single vendor. Another problem is that file formats do not allow any recovery. Take something as simple as a jpg image. A few bytes flip in the header and the whole image can be lost unless you have some complex recovery software. I once worked on a project where multiple copies of the header were written for a file in case the header got corrupted. The problem is the assumption that most standards bodies make is that that the header will never get corrupted or they would have multiple copies. Maybe someday everyone will finally come to the realization that digital data will not be and will never be 100 percent reliable and deal with the problem.



Labels: high availability, data integrity, Storage

posted by: Henry Newman

Archival Data vs. Archive

There needs be more emphasis on a tier of storage not accessed much that must be constantly available. Archival data is data that you need to keep but you might not access either for long periods of time or ever. This type of data exists in many fields from Sarbanes-Oxley compliance data, to weather forecasting, to Landsat satellite images, to medical records, to my childhood pictures. None of this data is likely needed every day or even every week, but we still need to keep it.

It would be nice if there were a way to keep this data safe and secure (whether I am a home user or a company large site or government site) that scales, is easy to manage, and has a reasonable price. Archives today require people with HSM skills and a significant investment in hardware and software. But there is NO guarantee your data is safe, as no one I am aware of calculates the reliability of the data in the archive, which I know from firsthand experience is not easy to determine. I believe this is a problem ripe for solving. There are, of course, a number of problems that need to be addressed, such as the reliability that is the liability. Lawyers will have a field day in case of data loss, and there will be some data loss. No storage medium is 100% reliable, even with multiple copies.

The more I think about this problem, the more I think that we need to reset expectations for those that archive data and expect and demand it back from the archive.


Labels: data storage, archive

posted by: Henry Newman

Disk, Tape and Flash Density

I recently read an interesting paper on expected densities of various technologies.

The paper looks at areal density growth and the potential for growth over the next few years. Contrary to what many vendors will tell you, it is clear from the paper that only tape has the potential to continue on the growth curve that it has been on. The HDD and NAND growth rate of areal density will likely drop to 20 percent per year. Areal density translates into device density. The points in the article are some of the points I have made in the past, but this is a detailed scientific study of the underlying issues and technologies that translate into areal density from materials to head design.

Sadly, the paper, and its ideas, have not gotten the press that is should have. Is this because some of the storage vendors do not want you to hear about it (and, yes, I believe that some conspiracy theories really are not theories but facts)? Or is it that the computer trade press does not want to get down and dirty with complex technological facts? Or is there some other reason? I am not sure that the computer press covers issues that go against the grain of marketing dollars. I am lucky very lucky that I am allowed to address controversial topics without fear of editorial reprisal. I am not so sure others are as lucky.

To end this entry, please repeat after me, "Disk and tape are not dead, disk and tape are not dead. Flash will not replace disk and tape."

Labels: Flash, backup, tape, disk, density, Storage

posted by: Henry Newman

Fibre Channel, Infiniband and 10 Gb Ethernet

I have been thinking for a number of years that slowly but surely Fibre Channel (FC) will be taken over by 10 GbE and that even longer term Inifiniband (IB) will prevail.

I really mis-predicted what the market would do back in my 2008 predictions. I had no idea the economy would tank so far so fast. FCoE did not take over the market, as I predicted, and cost of Ethernet ports did not go down nearly as quickly as I had expected.

But that was then, and this is now. I recently looked quickly at pricing on Froogle for some small FC, 10 GbE and IB switches. Here are the prices. Your mileage may vary, of course.

  • 48 port 10 Gb E switch is ~$7,599 ($158 per port)
  • 32 port FC switch ~$17,999.00 ($562 per port)
  • 36 port QLogic ~$5000 ($138 per port)

This is just switch costs without cables. IB and FC adaptors are far more than 10 GbE already.

My understanding from talking with some people in the industry is that large enterprise Ethernet switches cost per port is far less than large IB switches per port. Also remember that Ethernet has far more functionality than IB, but IB does have lower latency. Does latency really matter for big storage? The latency in the datapath to the drive is orders of magnitude more than the latency difference between IB and Ethernet. Ethernet has been and, at least for the foreseeable future, will be a commodity. Neither FC nor IB ever became a commodity and never, I believe, will be. Hence, the price, no matter which way you look at it, will always be higher. As of today, there are only two IB vendors in business. How can IB become a commodity technology and compete with Ethernet in the long term on price? I think we all know the answer.




Labels: Fibre Channel over Ethernet, 10 GbE, fibre channel, EtherNet, InfiniBand, network protocol

posted by: Henry Newman

More on Standards Stagnation

Why do the IETF, ANSI T10, T11 and T13 and a whole host of other standards bodies make progress in term of updating standards for new technology while we have archaic standards for user level I/O and file systems? I asked myself that question and have been pondering the potential reasons why. Here is a list of possible reasons why other standards make progress and user level I/O is stuck in the past:

  1. New standard are required for new hardware. The ANSI T1X groups are outgrowths of hardware development of new technologies that vendors need to sell. There is a large group of vendors trying to develop these new technologies from Fibre Channel to SAS to SATA, from disk and tape drives to connectivity.
  2. The IETF is trying to address many factors including performance, security, and hardware changes and involves, and it has hundreds of involved organizations.

User level POSIX I/O standards are part of the operating system and the C library. These standard used to be controlled by, if I remember correctly, in order: Bell Labs, UNIX System Laboratories, UNIX International, the OpenGroup. There were, again if I remember correctly, at most 10 vendors that wrote operating systems and worked on standards. Today, there are far fewer (my list is MVS, AIX, Solaris, Windows BSD/MacOS, and, of course, Linux). In reality, in terms of volume, there are three today: Windows, Linux and BSD/MacOS. Since the OS vendors are the ones that have to do all the work to develop the standards, make the changes and do the testing, what is their incentive, especially in hard economic times? There is no incentive without a significant reason, and the user community has not demanded it.

Labels: standards, POSIX

posted by: Henry Newman

The Digital Preservation Disconnect

I help put together a meeting on digital preservation for a U.S. Government Agency every year. The goal of the meeting is to bring together the industry with archivists to discuss technology issues. One of the major areas of disconnect is comparing what the two groups think about in terms of data loss. Technology people talk about data loss in terms of 9's, just like we talk about availability in terms of 9's. Preservationists talk about data loss as "we do not accept any data loss." If you lost a book, and that was the only copy, then the book is lost forever. Here is how I think about data loss.

Henry Newman's view of data loss

Clearly, no one can afford even 15 9s of data integrity for large data volumes. Data loss, given that storage technology is not perfect and we are not paying for it to be perfect, is inevitable. The digital preservation community, and most importantly its management, must accept this or alternatively pay for the number of 9s that its members want. The problem is that the storage community does not have a consistent way of talking about data integrity; nor do the few vendors that provide it.

As we move to develop common nomenclature within the storage community to discuss these issues, we must also move toward a way to talk about these issues externally. No one has any idea what the bit-wise data integrity is for data moving from a host to storage and back, and given all of the hardware and software involved, it is really hard to test for end-to-end integrity.

As more and more data gets archived, and both sides have unrealistic and different expectations for what will happen, I suspect we are all cruising for a bruising. The first vendor that loses lots of high-profile data will get all the bad press, but it is not likely totally their fault; it is everyone's.

Labels: data protection, archive, Storage

posted by: Henry Newman