There are three main problems with SPEC benchmarks as a measure of performance. First, they include a set of general purpose programs motivated by typical software engineering and design environments. We believe that extrapolations from the performance on these programs to specific application domains would be dubious. SPEC reduces the performance of a system on a set of different programs to just two numbers-SPECint and SPECfp. We believe that this methodology is inadequate and that benchmarks consisting of a set of applications that represents a typical workload should exist in each domain. These benchmarks will not necessarily be valid for other domains. Secondly, SPEC benchmarks are supposed to measure CPU performance and hence they are compute-intensive. There are application domains, CAD tools for VLSI design being a prime example, where a lot of applications are memory-intensive and the memory organization of a machine is an important factor in determining the tool response time. Hence, a meaningful benchmark suite should contain applications that exercise the memory hierarchy. The third problem with SPEC numbers is that only one input is used for each application in the benchmark suite. However, the relative performance of computers is not monotonic in the size of the input. To come up with meaningful metrics for the performance of computer systems, it is necessary to take the variation of relative performance with increasing problem size into account.
With the proliferation of CAD tools to support synthesis and verification of electronic circuits, performance evaluation of computers for CAD applications is becoming important. In this work, we take a set of CAD applications spanning the whole ASIC design flow from high level specification to physical design, and benchmark five architectures against them. Each application is fed by a sequence of input designs of increasing size. A direct consequence of using different input sizes for the applications is that memory organization and its performance will play an important role in determining the performance of the machines. Small problems which fit in the cache will be relatively independent of the on-board memory performance. However, for large problems, the performance of the memory organization will strongly affect the overall architecture performance. We try to identify features of the memory organization of the machines that lead to differing performance characteristics for different applications and input sizes.
Other benchmarks have been designed for specific application areas,
e.g., SFS benchmark to measure performance of file servers,
DebitCredit for benchmarking transaction processing
system [2], self-scaling benchmark for I/O performance
analysis [3], and PAS [4] and PERFECT [5]
for scientific computing, to name a few. However, to the best of our
knowledge there has been no work in benchmarks in the domain of CAD
applications. In this work, we propose a static benchmark
consisting of a set of CAD applications with single user
response time as our performance measure. We believe that single user
response time is more appropriate because human designers use these
applications and there is a direct correlation between tool response
time and productivity.
The rest of the paper is organized as follows. In Section 2, we describe our methodology including the selection of applications, architectures and input designs. We present and analyze our experimental results in Section 3.