I just spent a week evaluating a number of proposals. It got me thinking about what skills are needed and learning required to work on an evaluation team. Of course, every evaluation is slightly different, but there is, I think, a set of team skills. First and foremost, it is critical everyone on the team be able to communicate verbally and have reasonable writing skills. It really is of no value to put someone on the team that comes to the evaluation meeting and says nothing and has not provided some written information before the meeting. Second, I really think the teams need to be divided into the component skill sets:
- Applications and users: You need to have people that will be working on the system with the proposed applications, whether they be users or developers. That can be database, scientific, a CRM application or VMware.
- Administration: Security, systems and network people must be involved in the decision. System and network certification for the security of the operating system must be considered along with the whole administrative model.
- System architects: People that understand low-level performance implications of things such as memory bandwidth, PCIe bus bandwidth and configuration, and I/O issues as they apply to the application requirements.
- Performance: Part of the team needs to evaluate the performance of the proposed environment on the real applications. I have seen many cases where vendors propose almost all of the system basic hardware components, but say a different compiler or file system or other minor detail.
Part of what you are evaluating is, does the vendor understand your environment and the industry, and does the bid have the right stuff to meet your needs.