ELEC2300 Computer Organization
Lecture 4: Performance
Evaluation
qProfessor George Yuan
qOffice: Rm. 2527
qEmail:[email protected]
Note: some of the slides are adapted from Computer Organization and Design.
Copyright 1998 Morgan Kaufmann Publishers and Notes of Prof. Pattersons CS152
Class, Copyright 1997 UCB.
OUTLINE
qWhat is the computer performance?
qHow to evaluate the performance?
ELEC2300 Computer Organization Fall 2013
Page 2
Which of these airplanes has the best performance?
4 types of airplanes fly between Hong Kong & Shanghai
(distance: D mi.)
Airplane
Boeing 737-100
Passengers
Range (mi)
101
630
Boeing 747
470
BAC/Sud Concorde
132
Douglas
DC-8-50
Time
to perform146
the
Speed (mph)
598
4150
610
4000
1350
8720
544
(Execution
Time)
task
execution time, response time, latency
D
L=
S
Tasks per day, hour, week, sec, ns. ..
1 S
1
T=
C = S C
throughput, bandwidth
D
D
Latency and throughput often are in opposition
ELEC2300 Computer Organization Fall 2013
Page 3
Example
Execution time of Concorde vs. 747:
vConcorde is 1350 mph / 610 mph = 2.2 times faster
q Throughput of Concorde vs. 747:
vBoeing is 286700 pmph / 178200 pmph = 1.6 times
faster (470*610=286700, 132*1350=178200)
q Conclusions:
vConcorde is 2.2 times faster in terms of flying time.
v747 is 1.6 times faster in terms of throughput.
q
ELEC2300 Computer Organization Fall 2013
Page 4
Execution Time vs. Throughput
q Execution time
v How long does it take for my job to run?
v How long does it take to execute a job?
v How long must I wait for the database query?
q Throughput:
v How many tasks can the machine run at once?
v What is the average execution rate?
v How much work is getting done?
qComputer upgrade:
1.P3 -> P4
2.1 P3 -> 2 P3
qWe will focus primarily on execution time for a
single job.
ELEC2300 Computer Organization Fall 2013
Page 5
Definitions
qFor computer study,
1
performanceX =
execution
time X
" X is n times faster than Y" means
n = performance X = execution time Y
performance Y execution
time X
Problem:
vmachine A runs a program in 20 seconds (1 program/20
sec)
vmachine B runs the same program in 25 seconds (1
program/25 sec)
ELEC2300 Computer Organization Fall 2013
Page 6
Execution Time
qElapsed time or response time
vcount everything (disk and memory accesses, I/O , etc.)
va useful number, but often not good for comparison purposes
qCPU time
vDoes not count I/O or time spent running other programs
vcan be broken up into system time, and user time
qOur focus: user CPU time
vtime spent executing the lines of code that are "in" our program
vSystem CPU time: time the CPU spends executing system
(kernal) code in order to run your program, such as, reading
files, moving information into and out of virtual memory, etc.
performanceX =
1
user CPU
time X
ELEC2300 Computer Organization Fall 2013
Page 7
CPU Time Measurement: Clock Cycles
qInstead of reporting execution time in seconds, we often
use cycles
seconds
cycles seconds
=
program program cycle
qProcessor runs machine instructions based on clock
clock cycle time
time
qclock rate (frequency) = cycles per second (1 Hz. = 1
cycle/sec)
A 200 Mhz. clock cycle time is
ELEC2300 Computer Organization Fall 2013
Page 8
Relating the Metrics
qCPU time for a program
CPU time = CPU clock cycles * clock cycle time
= CPU clock cycles/clock rate
qCommon ways to improve performance
(i.e. shorten CPU execution time):
vReduce number of required CPU clock cycles for
a program
vShorten clock cycle time (i.e. increase clock rate)
ELEC2300 Computer Organization Fall 2013
Page 9
Example-Problem
q Description:
vA program takes 10 seconds to run on a 400 MHz
machine (computer A). We want to design a faster
machine (computer B) that can run the same program
in 6 seconds.
vThe increase in clock rate affects the rest of the CPU
design, causing machine B to require 1.2 times as
many clock cycles as machine A for the program.
q Problem to solve:
vWhat clock rate should machine B have?
ELEC2300 Computer Organization Fall 2013 Page 10
Example - Answer
ELEC2300 Computer Organization Fall 2013 Page 11
Cycle Number Calculation
qCPU time for a program
CPU time = CPU clock cycles * clock cycle time
= CPU clock cycles/clock rate
compiler
program
assembly program
assembler
compiler
Instruction #
machine instructions
ISA
processor
clock cycles/instruction (CPI)
Cycle # = Instruction # CPI
ELEC2300 Computer Organization Fall 2013 Page 12
Cycles Per Instruction
qWrong assumption:
v# of CPU clock cycles in a program = # of instructions in the
program,
qActual situation
vFor some processors, some instructions may take more cycles
than the others:
E.g. multiplication takes more cycles than addition
Floating point operations takes more cycles than integer
operations
Memory access takes more cycles than accessing registers
vConclusion: not all instructions require the same # of cycles to
execute.
qCycle per instructions (CPI) an average number of
clock cycles that each instruction in a program takes to
execute.
ELEC2300 Computer Organization Fall 2013 Page 13
Cycles Per Instruction (CPI)
qDefinition (for a given program):
CPI = (CPU clock cycles)/(instruction count)
qA program has the same instruction count on two
different implementations of the same instruction set
architecture, but it may have different CPIs (because an
instruction may require different numbers of clock cycles
on different implementations). If the number of clock
cycles for a program is known, knowing either the
instruction count or the CPI can determine the other.
qCPI provides a measure for comparing implementations.
qInstruction count can be measured using software tools
or simulators.
ELEC2300 Computer Organization Fall 2013 Page 14
Cycles Per Instruction
qLet there be n different instruction classes
(with different CPIs). For a given program,
suppose we know:
vCPIi = CPI for instruction class i
vCi = # of instruction of class I
qCPU clock cycles = CPI * instruction count. It
can be generalized to
n
CPU _ clock _ cycles = (CPI i Ci )
i =1
and
i =1
i =1
CPI = (CPI i Ci ) / Ci
ELEC2300 Computer Organization Fall 2013 Page 15
CPI Example
qSuppose we have two implementations of the
same instruction set architecture (ISA)
qFor some program, machine A has a clock cycle
time of 1 ns (1 GHz) and a CPI of 2.0. Machine
B has a clock cycle time of 2 ns (500MHz) and a
CPI of 1.2. Which machine is faster for this
program, and by how much?
qIf two machines have the same ISA which of our
quantities (e.g., clock rate, CPI, execution time, # of
instructions, MIPS) will always be identical?
ELEC2300 Computer Organization Fall 2013 Page 16
Example - Solution
ELEC2300 Computer Organization Fall 2013 Page 17
Relating the metrics
qFor a given program X running on a machine A
Time =
seconds
# of instructions
=
program
a program
# of clocks
second
*
*
# of instructions
clock
= instruction count * CPI * clock cycle time
= instruction count * CPI / clock rate
qThe only complete and reliable measure is CPU execution
time
qOther measures are unreliable. E.g. changing the
instruction set to lower the instruction count may lead to a
larger CPI or an organization with a slower clock rate.
Either case can offset the improvement in instruction count.
ELEC2300 Computer Organization Fall 2013 Page 18
Example Comparing Code Segments
q Description
vA particular machine has the following hardware facts:
Instruction class
A
B
C
CPI for this instruction class
1
2
3
vFor a given C++ statement, a compiler designer considers two
code sequences with the following instruction counts:
Code sequence
1
2
Instruction counts for instruction classes
A
2
4
B
1
1
C
2
1
q Problem to solve
vWhich code sequence executes the most instructions? Which is
faster? What is the CPI for each sequence?
ELEC2300 Computer Organization Fall 2013 Page 19
Example - Answer
ELEC2300 Computer Organization Fall 2013 Page 20
A misleading measure - MIPS
qThere are some performance measures that are
famous among computer manufacturers and
sellers but are misleading!
qMIPS (million instructions per second)
(meaningless indication of processor speed)
vMIPS = (instruction count)/(execution time * 106)
vMIPS depends on
Instruction set (instructions have different capabilities)
Program
vMIPS can vary inversely with performance
vPeak performance
ELEC2300 Computer Organization Fall 2013 Page 21
Some Processors in MIPS
Processor
IPS
Year
Motorola 68000
1MIPS @ 8MHz
1979
Intel 386DX
8.5MIPS @ 25MHz
1988
Intel 486DX
54MIPS @ 66MHz
1992
PowerPC G2
35MIPS @ 33MHz
1994
Intel Pentium Pro
541MIPS @ 200MHz
1996
ARM 7500FE
35.9MIPS @ 40MHz
1996
PowerPC G3
525MIPS @ 233MHz
1997
Zilog eZ80
80MIPS @ 50MHz
1999
Intel Pentium III
1354MIPS @ 500MHz
1999
AMD Athlon
3561MIPS @ 1.2GHz
2000
Pentium 4
9726MIPS @ 3.2GHz
2003
ARM Cortex A8
2000MIPS @ 1.0GHz
2005
Xbox360 IBM Xenon Triple Core
6400MIPS @ 3.2GHz
2005
AMD Athlon 64 3800+ X2(Dual Core) 14564MIPS @ 2.0GHz
2005
Intel Core2 Extreme QX6700
2006
57063MIPS @ 3.33GHz
ELEC2300 Computer Organization Fall 2013 Page 22
Another misleading measure - MFLOPS
qMFLOPS (million floating-point operations per second):
vMFLOPS =
(# of floating point operations)/(execution time * 106)
vMFLOPS considers only floating-point operations
(addition, subtraction, multiplication, or division
operation applied to a number in a single or double
precision floating-point representation).
vMFLOPS depends on:
Floating-point operation
(e.g., addition and multiplication differ in complexity)
Program
vMeaningless if there is little or no floating-point
arithmetic.
ELEC2300 Computer Organization Fall 2013 Page 23
MIPS example
q Two different compilers are being tested for a 100 MHz. machine
with three different classes of instructions: Class A, Class B, and
Class C, which require one, two, and three cycles (respectively).
Both compilers are used to produce code for a large piece of
software.
vThe first compiler's code uses 5 million Class A instructions, 1
million Class B instructions, and 1 million Class C instructions.
vThe second compiler's code uses 10 million Class A instructions,
1 million Class B instructions, and 1 million Class C instructions.
q What are the execution times for each sequence?
q What is the MIPS index for this processor based on the two testing
sequence?
ELEC2300 Computer Organization Fall 2013 Page 24
Summary
qSome related terminology:
vclock, clock cycle, cycle
vclock cycle time, cycle time (seconds, us, ns)
vclock rate, cycle rate (Hz, MHz)
vCPI (cycles per instruction)
vMIPS (millions of instructions per second)
qPerformance is determined by the execution time
qExecution time calculation:
Execution Time = instruction count * CPI * clock cycle time
= instruction count * CPI / clock rate
ELEC2300 Computer Organization Fall 2013 Page 25
OUTLINE
qWhat is the computer performance?
qHow to evaluate the performance?
ELEC2300 Computer Organization Fall 2013 Page 26
Benchmarks
q Execution time calculation:
Execution Time = instruction count * CPI * clock cycle time
= instruction count * CPI / clock rate
q Benchmark: a set of specially designed programs to test the
performance of a computer
q Performance best determined by running a real application
vBenchmarks are application specific
CPU performance, graphics, high-performance computing, objectoriented computing, Java applications, client-server models, mail
systems, file systems, Web servers.
q SPEC (System Performance Evaluation Cooperative)
vcompanies have agreed on a set of real program and inputs
vvaluable indicator of computer performance
Processor (ISA implementation) + compiler
ELEC2300 Computer Organization Fall 2013 Page 27
SPEC 89
q Compiler enhancements and performance
800
700
SPEC performance ratio
600
500
400
300
200
100
gcc
espresso
spice
doduc
nasa7
li
eqntott
matrix300
fpppp
tomcatv
Benchmark
Compiler
Enhanced compiler
ELEC2300 Computer Organization Fall 2013 Page 28
SPEC CPU2000
qSPEC ratio
vReference: Sun Ultra 5_10 with a 300MHz
processor
qCINT2000, CFP2000
vGeometric mean of SPEC ratios
ELEC2300 Computer Organization Fall 2013 Page 29
SPEC CPU2000 Benchmarks
ELEC2300 Computer Organization Fall 2013 Page 30
SPEC CPU2000 ratings
ELEC2300 Computer Organization Fall 2013 Page 31
Amdahl's Law
Execution Time After Improvement =
Execution Time Unaffected +
( Execution Time Affected / Amount of Improvement )
Example:
"Suppose a program runs in 100 seconds on a machine, with
multiplication responsible for 80 seconds of this time. How much do we
have to improve the speed of multiplication if we want the program to run
4 times faster?"
How about making the program 5 times faster?
Principle: Make the common case fast
ELEC2300 Computer Organization Fall 2013 Page 32
Example
q Suppose we enhance a machine making all floating-point instructions
five times faster. If the execution time of some benchmark before the
floating-point enhancement is 10 seconds, what will the speedup be if
half of the 10 seconds is spent executing floating-point instructions?
q We are looking for a benchmark to show off the new floating-point
unit described above, and want the overall benchmark to show a
speedup of 3. One benchmark we are considering runs for 100
seconds with the old floating-point hardware. How much of the
execution time would floating-point instructions have to account for
in this program in order to yield our desired speedup on this
benchmark?
ELEC2300 Computer Organization Fall 2013 Page 33
Remember
qPerformance is specific to a particular program
vTotal execution time is a consistent summary of performance
qFor a given architecture performance increases come
from:
vincreases in clock rate (without adverse CPI affects)
vimprovements in processor organization that lower CPI
vcompiler enhancements that lower CPI and/or instruction count
qPitfall: expecting improvement in one aspect of a
machines performance to affect the total performance
qYou should not always believe everything you read!
Read carefully!
ELEC2300 Computer Organization Fall 2013 Page 34
