I was on a panel at the IEEE Mass Storage conference last week, and the main topic was how to converge HPC applications and cloud applications. You might ask, what do HPC applications and cloud applications have to do with each other? The panelists all believe that they intersect in the area of data analysis.
No, we are not analyzing the same data, but the idea of big data analysis has been a common theme in HPC for a long time. Whether it has been analyzing weather forecast predictions or car crash simulations and comparing them to crashing real cars, HPC has always had a big data analysis requirement. As people consider data consolidation to either public or private clouds or even locally changing data into actionable information, they are facing the next big challenge in the IT world in my opinion.
As all of you know, computational performance gains have far outpaced storage performance gains even if you add NAND flash into the equation. If you read my last blog entry, you realize that the cost for NAND will not be dropping to the point we can put all of our data on NAND. This means there is a need for the job description that I would call Data Analysis Architect. The person would understand all of the data analysis techniques, from MapReduce to graph analysis, and statistical methods and would have a good understanding of how to lay out the data so it could be processed to create actionable information for the organization. It might be something like a 15-day weather forecast for a commodities trader or farm insurance broker, or what combination of things work best to bring people into a retail or big box store. Who knows what information can be found from the various nuggets of data if it can only be processed fast enough.