Henry Newman's Storage Blog Archives for March 2014

Doubts about Gartner's Analysis of Enterprise Arrays

As you might have seen from lots of press releases, Gartner has released their view of enterprise arrays and for the first time, mid-range arrays. For the enterprise arrays the Gartner analysts have weighed in on best to worst, based on their criteria.

As least in the trade publications I have read (no way I am going buy the reports), there seems to be not much, if any, discussion of out-of-the-box experience or operational experience over a significant period of time or lots of other things.  We do know that Gartner says that performance does not matter anymore for enterprise arrays because of flash. 

To me this is clearly proves as clear as day that the reports are seriously flawed.  Nowhere is there a discussions of the backend performance (from array cache to storage) of any of the arrays. I know for sure that some of the 12 arrays on the enterprise list have far more performance and some have, well shall we say, abysmal performance.  

Gartner picked five use cases:

1. Consolidation


3. Server virtualization and VDI

4. Analytics

5. Cloud

I could see where performance would not matter in the cloud as the network latency is likely the dominate factor, but I cannot understand the statement for the other applications.

So here are a few of my questions, in no particular order:

1. Do these analysts every get on one of the arrays they are reporting on? Do they test it from the time it gets off the loading dock to, say, a month or two from then?

2.  Do these analysts look at the underlying architecture of the storage including things like Memory bandwidth of cache? Design of the front and back end connectivity to cache (think PCIe buses and channels)?

3. How are reliability, manageability and usability measured, and how does it fit into the rating?

There can only be one winner in any analysis but there has to be a clear understanding of what you are analyzing and why – and why it matters.  At least from the press reports I am seeing I am more confused than ever on the Gartner analysis methodology. 

Photo courtesy of Shutterstock.

Labels: RAID, Gartner, array, disk array

posted by: Henry Newman

Data Storage: Metadata Everywhere (And Not a Drop to Drink)

This article's catchy title comes from The Rime of the Ancient Mariner by Samuel Taylor Coleridge, but a co-worker suggested I might want to title it "Metadata everywhere causing everyone to drink" instead.

The word "metadata" is used to describe lots of different things. As a storage guy, my introduction to metadata was in discussing file systems and requirements for metadata. Ask a librarian, and metadata is something totally different. Ask a database person, and again you will get a different answer. Look at a jpg header and you have different metadata about the file, not the file system. "Metadata" might be one of the most over used and confusing terms being used today. It might as well be a pronoun like “that,” which can mean just about anything any time.

I want to talk about a few types of metadata. I'll share some of my thoughts on what needs to happen in the future and what I think will happen and why.

High-level storage systems metadata

High-level storage systems metadata" is a term I just made up. The reason I used the words "high-level" is that I view low-level storage systems metadata as information about how the controller views the underlying storage under its control. Low-level information includes information on disks in RAID groups, LUN information and data on virtualization of the drives, replication and lots of other stuff.

To me "high-level" implies that users are going to access information via an interface, file system, protocol or REST. We used to just call it file system metadata. That term had specific meanings, connotations and implications for about 25 years. For example, file system metadata had to support POSIX file system standards. It included things like user id (UID), group id (GID), access time (atime), create time (ctime) and other categories that are part of standard POSIX, plus additions for things like NFS and access control lists (ACLs). All of this information for file systems was stored in the inode, whether that was a local file system or on a NAS device. The specific minimal set was defined by the POSIX standards, and NFS added things like ACLs. In addition, there were extensions some vendors added using POSIX-extended attributes for things like high hierarchical storage management. The inode also stored the location within the file system on disk. Different vendors implemented things a bit differently, but all of this was in the inode or an extended attribute that was viewed as part of the inode.

Well that was then; this is now. With new interfaces such as REST, POSIX-compliant and like inodes are just one of the metadata components that users might see.

The problem with POSIX inodes and the standard was that it was not very extensible. Of course, extensibility was theoretically built in with POSIX extended attributes. But as we all know, theory and reality are often different. Issues arose because each vendor might—and often did—use extended attributes, but the attributes between vendors and various file systems were never agreed upon. About a decade ago, a group of people in the US government and HPC tried to extend POSIX and were rejected by the OpenGroup which controls POSIX. Extended attributes were defined only for a specific file system. POSIX, therefore in my opinion, is pretty limited for adding modern metadata information.

REST interfaces, on the other hand, are easy to extend compared to file systems. Many REST implementations are accomplished with databases, and adding a new bit of information is pretty easy. The mapping of objects to their storage locations is often similar to what is done in standard file systems today, but developers have learned a great deal over the last 25 to 30 years. There are now lots of new things users can provide per object like checksums and field. All of this gives REST a big advantage, but from what I have seen, there is not a great deal of coordination so that everyone gets the same per object metadata and you can easily move from one object store to another.

Per file/object metadata

Lots of file types (soon they might be named "object types") have their own metadata which describes the file or object. For example, a jpg has lots of information describing the file. For medical images, there are Dicom files, and the format is controlled by National Electrical Manufacturers Association (NEMA), many of whose members are vendors that make scanning equipment like MR, CT and X-rays. In weather, there is GRIB (Gridded Binary or General Regularly-distributed Information in Binary form) which has been agreed upon by the World Meteorological Organization. Another example would be oil and gas exploration companies, who also have standard formats, and yet another example would be the electric power grid community.

Labels: REST, metadata, POSIX, Storage

posted by: Henry Newman

Data Storage Mysteries: What Happened to MAID?

It was at little less than a decade ago that MAID (massive array of idle disks) was making headlines almost daily.  The press and industry were all over it. Our favorite two analyst companies were drawing charts and graphs telling us that MAID was going to change our world.

Well, we all know that that did not happen.  I think there are a number of reasons why MAID did not change our world. These include, but are not limited to:

· Power savings were not as great as expected given that the disk controllers needed to be powered up and that was (from what I could tell) about 30% of the power for the storage.

· Disk drive reliability and rebuilding drives for failed LUNs was a big issue, as the device needed to be powered up for rebuild.

· Software management for backup and archive software used the powered up devices rather than powering up new devices, along with issues for file systems.

· MAID cost per TB never achieved the expectations set by industry.

So what did we learn from MAID? We learned that power management is far more difficult and complex than anyone had thought, and powering on and off drives up to 10s of times per day caused a number of reliability issues.  

One of the things we should have learned – and we still have not – is that the industry analysts are not that great at predicting the future.  MAID basically died and one of the reasons was that hard drive density was increasing pretty quickly during that period of time for SATA drives, and that, combined with the economics of MAID, did not make much sense. That was not news given the public roadmaps.

The power for 1 PB of drives using 8+2 RAID-6, 4 TB drives, adding 30% for each drive for the controller and .10 per KW hour is using average power per drive $3,464.23 and $2,989.01 using idle power per drive.Clearly MAID never did make much sense.

Why were we listening to the hype?

Labels: data storage, MAID, drives

posted by: Henry Newman

The Need for Realistic Benchmarking Protocols

Many of us today are accessing storage via appliances and NFS or CIFS, yet neither of these protocols have seen significant changes over the last few decades. pNFS has been a promise for almost a decade and, while there have been some implementations, it has not caught on in terms of market share for a whole bunch of reason, not the least of which is the lack of parallel file system support and development in this area, except for one vendor.

The most recent benchmarks data I have seen for the SPECNFS (and other similar benchmarks) has been done using all flash systems, which we all know is unrealistic. I have a number of customers that copy their files from the NFS storage to their local disk and then operate on the data and then copy the file back to the NFS storage. They do this because the interactive operations they are doing on the information in the file is just too slow over NFS and even worse over CIFS, and from a productivity point of view doing this is a requirement.

It is impossible for these customers to do their work and still run over NFS or CIFS, and meet the organization’s productivity objectives. I think it is time for vendors and the industry to create realistic benchmark on realistic storage configurations that customers will buy for storage that is sold where connectivity is over NFS or CIFS. 

The customer in question cannot afford 500+TB of usable flash storage, and disk-based benchmarks using these protocols are not being done. NFS and CIFS protocols need to be fixed to address the small block random problem. And without realistic benchmarks with real world problems and data all we we are going to get is hype. I am sure that the customer I have is not alone with this type of data analysis problem, but industry seems to be moving away from this, not embracing the real requirements.

Photo courtesy of Shutterstock.

Labels: data storage, benchmarks, NFC

posted by: Henry Newman