Storage Vendors Still Do Not Get It

I saw this headline last week. "Avere shatters SPECsfs2008 NFS benchmark."

This is nothing against Avere specifically--just another example of the benchmark arms race that is out of control and really does not provide much value to buyers. Just so everyone knows, the company I run does not resell any hardware or software and we will not. We are vendor-neutral. Remember that when you are reading articles from other writers, but back to the facts.

The benchmark arms race will continue, and I believe it will continue to provide buyers no actual useful information. Somehow the buyer community must change the discussion. The problem is that the vendors control the benchmarks and the definition of benchmarks. Being the cynical conspiracy theorist I am, I believe that just like with the POSIX I/O standards that vendors control and do not want to change, vendors that control the I/O benchmarks do not want to change. Reasons that come to mind include:

1. That vendors know how to run the benchmarks and do not want to change anything given the costs. Oftentimes, for these complex benchmarks there is large learning curve that is needed to run the benchmarks and get the needed results on the vendor hardware.
2. Vendors now do not want to show scalability, as I believe it is in our interest but not their interest. When buying a storage appliance and looking at performance data, what really matters is that the product scales to meet the performance and density requirements that you have for the long run. Given the testing we have done, I know this is true for some products; remember the vendors control the benchmarks

We need a buyers manifesto that says we care about scalability, not absolute performance. No one is going to buy the exact configuration used by any vendor running SPECSFS, so why should we care about results?