.
1 Levels and Types of Benchmarks
Based on the levels of performance they measure, benchmarks can be grouped into two
levels:
Component-level Benchmarks
System-level Benchmarks
Based on their compositions, benchmarks can be categorized into types:
Synthetic Benchmarks
Application Benchmarks
2.2 Component-Level Benchmarks
Component-Level benchmarks test a specific component of a computer system, such as the
video board, the audio card, or the microprocessor. They are useful for selecting a
component of a computer system, that corresponds to a particular function. Instead of
testing the performance of the system running real applications, component-level
benchmarks focus on the performance of subsystems within a system. These subsystems
may include operating system, arithmetic interger unit, arithmetic floating-point unit,
memory system, disk subsystem, etc. Examples of component-level benchmarks include:
SPECweb96 - measures the web server performance
GPC - measures graphics performance for displaying 3-D images
2.3 System-Level Benchmarks
System-Level benchmarks evaluate the overall performance of a computer running real
programs or applications. These benchmarks are useful when comparing systems of
different architectures. They take each subsystem into account, and indicate the effect of
each subsystem on the overall performance. Examples of system-level benchmarks include:
SYSmark/NT 4.0 - measures the performance of computers running popular
business applications under Windows NT 4.0
TPC-C - measures the performance of transaction processing system like
2.4 Synthetic Benchmarks
Synthetic benchmarks are created by combining basic computer functions in proportions
that developers feel will yield an indicative measure of the performance capabilities of the
machine under test. These benchmarks try to match the average frequency of operations
and operands of a large set of programs. Functions included in synthetic benchmarks are
usually created artificially to match an average execution profile and this impairs their
credibility.
Synthetic benchmarks are component-level benchmarks, and they evaluate a particular
capability of a subsystem. For example, a disk subsystem performance benchmark may
combine a series of basic seek, read, and write operations involving varying numbers of
disk blocks of varying sizes.
When evaluating the results from synthetic benchmarks, the following rules should be
followed:
understand the composition of the benchmark,
appreciate the factors contributing to the results,
determine if the benchmark tests functions that are typical of your workload and
environment.
Examples of synthetic benchmarks include:
WinBench 97 - measures the performance of a PC's graphics, disk, processor, video
and CD-ROM subsystems in Windows environment
MacBench 97 - measures the processor, floating-point, graphics, video, and CD-
ROM performance of a MAC OS system
2.5 Application Benchmarks
Application benchmarks employ actual application programs. Developers of these
benchmarks include applications that they feel perform common functions from within a
particular industry segment or a class of products. These application programs are run by
a macro of program operations which attempts to model the way users operate their
system.
Most application benchmarks are system-level benchmarks, and they measure the overall
performance of a system. When an application benchmark is run, it tests the contribution
of each component of the system to the overall performance. These benchmarks are usually
larger and difficult to execute and are not useful for measuring future needs.
The major drawback is that application benchmarks are subject to the benchmark
developer's interpretation of a "typical workload". Examples of application benchmarks
include:
Winstone 97 - tests a PC's overall performance when running Windows-based 32-bit
business applications
SYSmark/NT 4.0 - measures the performance of computers running popular
business applications under Windows NT 4.0
