I want to do a bit more ranting about PCIe 4.0 plans. Having a doubling of performance between now and 2015 will stifle innovation. Maybe that is the plan--have everyone think you are happily moving down the PCIe yellow brick road and then do something different. Something must be done to improve communications between CPUs. It has become abundantly clear that building memory systems with more than 4 sockets get very expensive, given the engineering design costs.
However, there are many problems that require many CPUs. Things like search and index have used Hadoop and connected lots of cheap CPUs. But many complex scientific and engineering problems cannot decompose problems such that communications is not a barrier to getting the work done. You are not going to run a complex earthquake, atmospheric or other simulation today without high-speed communications. Doubling performance with IB by 2015 (my bet is at least 2016) is not doing to do it for advancing science, unless of course everyone makes some breakthroughs in the area of algorithms that do not require communications. I am thinking that some vendor has something up its sleeve, and presto--we will have high-speed communications outside of the PCIe framework. While search engines do not depend on this, scientific advance does.
I am hard pressed to believe that some of the major vendors have not realized that and are holding back to see if they can gain some market advantage. Since one of the Chinese supercomputers had its own interconnect not requiring PCIe, it is clear to me that someone will get it. I cannot believe that the major CPU vendors do not. The new IBM P775 and current Cray XE/6 systems have addressed this problem