We strongly believe that reporting performance as a single number is inadequate for benchmarking machines. Even in this very restricted field of CAD applications we found extreme variation of performance of machines depending on the application as well as the input size. We feel this vindicates our viewpoint that computer system performance benchmarks should be domain specific. Moreover, they should report results on a range of inputs of varying sizes. We hope this study can provide some guidelines for computer system designers who are vying for the lucrative CAD market, and to the users of such systems for evaluating the various options available to them. Clearly no one system will be suitable for all the applications and an effort to select a system on the basis of the expected workload should be made. We also found that performance of systems running CAD applications critically depends on memory and for some applications it depends solely on the memory performance. Hence a lot of design effort must be put in the memory organization.
We have presented a benchmark suite for CAD applications, which we believe is representative of the tools used in the industry at different levels of the design flow ranging from hardware compilation to physical design. We also presented performance results for five machines on inputs of varying sizes and highlighted the impact of input size on the performance.
The access to source code for uniform compilation restricted our application choice to public-domain programs. It would be interesting to benchmark these machines on industrial CAD tools. We would also like to use some industrial inputs for the applications. It would be instructive to repeat our experiments by using native compilers for all the architectures. We could not do this for reasons of availability and porting problems. We limited ourselves to the first and second level cache as far as assessing the performance of the memory organization was concerned. A study of the disk I/O for CAD applications would be interesting.