You might or might not have seen the article on HPC wire about IBM using a Violin Memory SSD to scan 10 billion files in 43 minutes. That is a scan rate of about 232,558,140 files per minute or 3,875,969 files per second. I was impressed. Today, top-end RAID controllers might support maybe 500,000 IOPS per second, and top-end RAID controller cards might do slightly less.
So what happens in the long term? Flash performance keeps going up far faster than RAID IOPS performance. Specialized flash devices from many vendors are far outstripping the performance of external RAID controllers and RAID controller cards, but the latter is far more expensive.
So given that this trend is going to continue (e.g., flash IOPS improving faster than RAID controller IOPs performance), what happens to RAID controllers? One of the models I look at is the storage hierarchy model you see with hierarchical storage management systems (HSM).
You have the disk for fast access, and some products can have multiple levels of disk and then, of course, tape. So my question is, in the long term, will we have specialized metadata appliances that are flash-based so that data moves to a different storage hierarchy, which will be RAID controllers?
Given we are moving back to the future with file systems separating data and metadata, which was first done, I think, on UNIX back by Cray Research in 1990, I believe we will be moving to hierarchical storage for file systems with data and metadata separation.