Application Interface For Data Storage: Change Needed

What part of the storage stack needs the most improvement to get the most performance and usability out of storage technology that is available? Some might say the storage controller, others the device, while a few might say the communications channel. 

My answer is the application interface to the storage, which for most applications has not changed in coming up on decades.  We have open(), read() and write() and the libc equivalent fopen(), fread() and fwrite(), all created in the 1970s and codified into a standard in the 1980s. 

Since that time, think about the performance changes we have seen: from 80 MB disk drives that run at 3 MB/sec the size of a washing machine that have an average seek and latency of 24 milliseconds, to high end PCIe and SAS attached flash drives supporting over 750 MB/sec transfer rate and with .1 milliseconds or less. And yet the application interfaces have not change to take advantage of these changes. 

There are no standardized I/O hints, no standardized policy management for the importance of the data, the expected usage or anything.  The standards we have POSIX were last changed in this area in any significant way in 1991 by adding asynchronous I/O.  Folks that was 23 years ago without a standards change.

Name anything else in our industry that has not had a standard change or improvement in 20+ years.  Here is my attempt to be diplomatic on this. This is behavior that is unbecoming to the engineering talent that works in this standards area.  Like it or not, this lack of change spells the future death of POSIX file systems. And we can say just, as we say to a cigarette smoker who gets lung cancer: we told you so.  You cannot have a standard that does not change with the times and expected to last.

Photo courtesy of Shutterstock.

Labels: data storage,application interface,Application Performance Management

posted by: Henry Newman

Henry Newman, InfoStor Blogger
by Henry Newman
InfoStor Blogger

Henry Newman is CEO and CTO of Instrumental Inc. and has worked in HPC and large storage environments for 29 years. The outspoken Mr. Newman initially went to school to become a diplomat, but was firmly told during his first year that he might be better suited for a career that didn't require diplomatic skills. Diplomacy's loss was HPC's gain.

Previous Posts