In my last blog post I commented that performance analysis is becoming a lost artand suggested that it will not be long before we go back to the future and begin working on performance analysis and code optimization instead of just buying hardware and more hardware. I do think we are going to have to go back and teach people how to optimize codes for the architecture being used.
The problem is, how do we develop those skills? Back when people were doing lots of code and system optimization, many of us were writing in assembly language. That taught us tough lessons about the hardware and how to efficiently use it. We need to teach people in our industry about hardware and how it works. Long ago, I was lucky enough to have been able to attend a class called Hardware Training for On-Site Analysts when I was working for Cray Research. It was taught by an EE who actually worked on the hardware design of the machine. I liked the class at the time, but over the years I came to realize how much that class meant to me and my career, giving me the tools to understand there is more than just software when doing performance analysis.
So how would we teach today's young people how to understand things like cache lines, memory bandwidth, NUMA and memory placement, SAS CRC errors, T10 DIF/PI and disk error recovery, just to name a few things? Honestly, I do not have a good answer. First of all, young people need to think that these kinds of things are important to learn, and almost all people today believe that you do not need to understand the hardware. I think we need to find a way to make understanding the hardware important again, but that will not be the case if the answer to all performance problems is buy more hardware.