Computer Architecture
A Quantitative Approach, Sixth Edition
Chapter 3
Instruction-Level
Parallelism and Its
Exploitation
Copyright © 2019, Elsevier Inc. All
rights Reserved 1
Un-pipelined Architecture
Unpipelined Start and finish a job before moving to the next
Fetch
Decode
Jobs
Execute
Time
2
Pipelined Architecture
Pipelined Break the job into smaller stages
F D X
F D X
F D X
Jobs
F D X
Time
3
5-Stage Pipeline
In order to enable pipelining we need to
hold or keep the input stable to each
stage and this require latching data and
control signals to each stage in the
pipeline → Pipeline Registers 4
Clocks and Latches
Stage 1 L Stage 2 L
Clk
• Unpipelined: time to execute one instruction = T + Tovh
• For an N-stage pipeline, time per stage = T/N + Tovh
• Total time per instruction = N (T/N + Tovh) = T + N Tovh
• Clock cycle time = T/N + Tovh
• Clock speed = 1 / (T/N + Tovh)
• Ideal speedup = (T + Tovh) / (T/N + Tovh)
• Cycles to complete one instruction = N
• Average CPI (cycles per instr) = 1 5
A 5-Stage Pipeline
6
A 5-Stage Pipeline
Use the PC to access the I-cache and increment PC by 4
P
C
PC+4
P
C
PC+4
P
C
PC+4
P
C
PC+4
7
A 5-Stage Pipeline
Read registers, compare registers, compute branch target; for now, assume
branches take 2 cyc (there is enough work that branches can easily take more)
8
A 5-Stage Pipeline
ALU computation, effective address computation for load/store
9
A 5-Stage Pipeline
Memory access to/from data cache, stores finish in 4 cycles
10
A 5-Stage Pipeline
Write result of ALU computation or load into register file
11
Introduction
• Pipelining become universal technique in 1985
• Overlaps execution of instructions
• Exploits “Instruction Level Parallelism”
Two main approaches:
• Hardware-based dynamic approaches
• Used in server and desktop processors
• Not used as extensively in PMP processors
• Compiler-based static approaches
• Not as successful outside of scientific applications
12
Instruction-Level Parallelism
• When exploiting instruction-level parallelism, goal is to
minimize pipeline CPI
• Pipeline CPI = Ideal pipeline CPI +Structural stalls +Data hazard
stalls + Control stalls
• **Parallelism with basic block is limited
• Typical size of basic block = 3-6 instructions
• Must optimize across branches
13
Instruction Dependences
• There are three different types of dependences
• Data dependence (True data dependence)
• Name dependence (Instructions using same register
names)
• Control dependence ( Branches )
•An instruction j is data-dependent on instruction
i if either of the following holds
– Instruction i produces a result that may be used by
instruction j
– Instruction j is data dependent on instruction k and
instruction k is data dependent on instruction i 14
Data Dependences
• Example of data dependence
Lp: fld f0,0(x1) //f0=array element
fadd.d f4,f0,f2 //add scalar in f2
fsd f4,0(x1) //store result
addi x1,x1,-8 //decrement pointer 8 bytes
bne x1,x2,Lp //branch if x1 ≠ x2
15
Instruction Dependences
• Dependencies are a property of programs
• Pipeline organization determines if dependence is
detected and if it causes a stall
• Data dependence conveys:
– Possibility of a hazard
– Order in which results must be calculated
– Upper bound on exploitable instruction level parallelism
• Dependencies that flow through memory locations
are difficult to detect
16
Name Dependences
• A name dependence occurs when two instructions
use the same register or memory location, called a
name, but there is no flow of data between the
instructions associated with that name
• Two types of name dependence
– An antidependence Write After Read or (WAR)
– Output dependence Write After Write or (WAW
17
Register Renaming
• Instructions with name dependence can execute
simultaneously or out of order if the registers are
renamed (register renaming)
• Renaming can be done statically at compile time or
dynamically by hardware at run time.
18
Control Dependence
• A control dependence determines the ordering of an
instruction i with respect to a branch instruction
if p1 {
S1;
};
if p2 {
S2;
}
• Instruction S1 is control dependent on p1 and S2 is
control dependent on p2
• Control dependence is preserved by implementing
control hazard detection that causes control stalls.
19
Control Dependence
• Can we move S1 after (if p2 ) or S2 before (if p1) ?
• Yes! but without affecting the correctness if p1 {
S1;
of the program };
if p2 {
S2;
• The two properties critical to program }
correctness are exception behavior and the data flow
add x2,x3,x4
beq x2,x0,L1
ld x1,0(x2)
L1:
• The load instruction may cause a memory
protection exception if moved before the branch
20
Control Dependence
• It is insufficient to just maintain data dependences
because an instruction may be data-dependent on
more than one predecessor add x1,x2,x3
beq x4,x0,L
sub x1,x5,x6
L: ...
or x7,x1,x8
• The or instruction is data-dependent on both the add
and sub instructions
• The data flow must be preserved.
• Speculation helps to lessen the impact of the control
dependence while still maintaining the data flow
21
Value liveness
• The property of whether a value will be used by an
upcoming instruction is called liveness
• What if we knew that the register destination of the sub
instruction (x4) was unused after the instruction labeled
skip? add x1,x2,x3
beq x12,x0,skip
sub x4,x5,x6
add x5,x4,x9
skip: or x7,x8,x9
• Then we can move the sub before the beq
• This type of code scheduling is also a form of
22
speculation, often called software speculation
Hazards
• Structural hazards: different instructions in different stages
(or the same stage) conflicting for the same resource
• Data hazards: an instruction cannot continue because it
needs a value that has not yet been generated by an
earlier instruction
• Control hazard: fetch cannot continue because it does
not know the outcome of an earlier branch – special case
of a data hazard – separate category because they are
treated in different ways
23
Structural Hazards
• Example: a unified instruction and data cache →
stage 4 (MEM) and stage 1 (IF) can never coincide
• The later instruction and all its successors are delayed
until a cycle is found when the resource is free → these
are pipeline bubbles
• Structural hazards are easy to eliminate – increase the
number of resources (for example, implement a separate
instruction and data cache)
24
Enabling and optimizing ILP
• To enable ILP we need to
– Detect data dependences either in software or hardware
– Insert stalls when ever needed for correct program result
– Flush pipeline when ever a branch is taken
• To optimize ILP we need to
– Minimize number of stalls needed for correct program result
• Know when and how the ordering among instructions may be changed
– Minimize flushing the pipeline
• Predicting branch outcomes.
25
Compiler Techniques for Exposing ILP
• Pipeline Scheduling
– Separate dependent instruction from the source instruction by
the pipeline latency of the source instruction
• Example
➢ C code:
for (i=999; i>=0; i=i-1)
x[i] = x[i] + s;
➢ Un-Pipelined RISC-V Code
Loop: fld f0,0(x1) //f0=array element x[i]
fadd.d f4,f0,f2 //add scalar in f2=s
fsd f4,0(x1) //store result
addi x1,x1,-8 //decrement pointer 8 bytes (per DW)
bne x1,x2,Loop //branch if x1≠x2
Where are the data dependencies in the above code? And of which type?
26
Compiler Techniques for Exposing ILP
➢ Pipelined RISC-V Code
Loop: fld f0,0(x1) Loop: fld f0,0(x1)
stall addi x1,x1,-8
fadd.d f4,f0,f2 fadd.d f4,f0,f2
stall stall
stall stall
fsd f4,0(x1) fsd f4,8(x1)
addi x1,x1,-8 bne x1,x2,Loop
bne x1,x2,Loop
Constrains
27
Compiler Techniques for Exposing ILP
• Loop unrolling
– Replicates the loop body multiple times, and adjusting the loop
termination code
– Unroll by a factor of 4 (assume # elements is divisible by 4)
– Eliminate unnecessary instructions
Loop fld f0,0(x1)
fadd.d f4,f0,f2
fsd f4,0(x1) //drop addi & bne
fld f6,-8(x1)
fadd.d f8,f6,f2
fsd f8,-8(x1) //drop addi & bne
fld f0,-16(x1)
fadd.d f12,f0,f2
fsd f12,-16(x1) //drop addi & bne
fld f14,-24(x1)
fadd.d f16,f14,f2
fsd f16,-24(x1)
addi x1,x1,-32
bne x1,x2,Loop
• Eliminating three branches and three decrements of x1 28
Compiler Techniques for Exposing ILP
• Pipeline schedule the unrolled loop
Loop: fld f0,0(x1)
fld f6,-8(x1)
fld f8,-16(x1)
fld f14,-24(x1)
fadd.d f4,f0,f2
fadd.d f10,f6,f2
fadd.d f12,f8,f2
fadd.d f16,f14,f2
fsd f4,0(x1)
fsd f10,-8(x1)
fsd
fsd
f12,-16(x1)
f16,-24(x1)
◼ 14 cycles
addi x1,x1,-32 ◼ 3.5 cycles per element
bne x1,x2,Loop
29
Compiler Techniques for Exposing ILP
❖ Determine that unrolling the loop would be useful by finding that the
loop iterations were independent, except for the loop maintenance
code
❖ Use different registers for different computations to avoid name
dependence.
❖ Eliminate the extra test and branch instructions and adjust the loop
termination and iteration code.
❖ Determine that the loads and stores in the unrolled loop can be
interchanged if they are independent, they do not refer to the same
address.
❖ Schedule the code, preserving any dependences needed to yield
the same result as the original code.
30
Compiler Techniques Limitations
❖ Loop overhead
❖ Amount of overhead that can be reduced decrease by each
additional unroll
❖ Code size limitations
❖ Increase in code size → possible increase in cache miss rate
❖ Compiler limitations
❖ Potential shortfall in registers --> register pressure.
31
Branch Prediction
❖ Basic 1-bit predictor:
❖ Predict not taken, just increment pc+4 (do nothing special)
T
T
N 0 1
N
32
Basic 1-bit predictor
▪ How basic 1bit predictor branch predictor behaves on
the following branch patterns?
▪ TTTTTTTTTTTNTTTTTTTTTTTTTTT…..
▪ NNNNNNNNNNNNTNNNNNNNNNNNNN….
▪ TNTNTNTNTNTNTNTNTNTNTNTNTNTNTN…..
33
Basic 1-bit predictor
▪ Assume 30% of instructions are branches, and 60%of
branches are mispredicted, calculate pipeline CPI if the
branch misprediction penalty is 2 cycles.
Pipeline CPI =
= 1 + %Branch Instructions %Branch Miss Prediction Rate Branch Miss Prediction Penalty
= 1 + 0.3 0.6 2 = 1.36
34
Resources
▪ Memory Timing
▪ https://www.hardwaresecrets.com/understanding-ram-timings/
▪ Memory Architecture
▪ https://en.wikipedia.org/wiki/Multi-channel_memory_architecture
▪ CS6810 Computer Architecture 87 Lectures by Rajeev
Balasubramonian
▪ https://www.youtube.com/playlist?list=PL8EC1756A7B1764F6
35
Resources
▪ HPCA short Lecture series on High Performance Computer Architecture
▪ Part 1 (161 Lectures)
▪ https://www.youtube.com/playlist?list=PLAwxTw4SYaPmqpjgrmf4-
DGlaeV0om4iP
▪ Part 2 (62 Lectures)
▪ https://www.youtube.com/playlist?list=PLAwxTw4SYaPkNw98-
MFodLzKgi6bYGjZs
▪ Part 3 (169 Lectures)
▪ https://www.youtube.com/playlist?list=PLAwxTw4SYaPnhRXZ6wuHnnclMLfg_y
jHs
▪ Part 4 (120 Lectures)
▪ https://www.youtube.com/playlist?list=PLAwxTw4SYaPn79fsplIuZG34KwbkYSe
dj
▪ Part 5 (149 Lectures)
▪ https://www.youtube.com/playlist?list=PLAwxTw4SYaPkr-
vo9gKBTid_BWpWEfuXe
36
How we implement a basic 1-bit predictor?
Branch PC
10 bits
Table of
1K entries
Each
entry is
a bit
The table keeps track of what the branch did last time
37
Basic 2-bit Branch Prediction
❖ Basic 2-bit predictor:
❖ For each branch:
❖ Predict taken or not taken
❖ Change prediction only if the prediction is wrong two consecutive times.
T T T
T
00 01 10 11
N
N N N
▪Check the following case assuming we start from 11 state:
TNTNTNTNTNTNTNTNTNTNTNTNTNTNTN…..
▪ We get 50% Correct prediction!
38
Basic 2-bit Branch Prediction
• For each branch, maintain a 2-bit saturating counter:
if the branch is taken: counter = min(3,counter+1)
if the branch is not taken: counter = max(0,counter-1)
• If (counter >= 2), predict taken, else predict not taken
• Advantage: few typical branches will not influence the
prediction (a better measure of “the common case”)
• Especially useful when multiple branches share the same
counter (some bits of the branch PC are used to index
into the branch predictor)
• Can be easily extended to N-bits (in most processors,
N=2)
• Prediction performance depends on both the prediction
accuracy and the branch frequency
39
Basic 2-bit Branch Prediction
Branch PC
10 bits Table of
1K entries
Each
entry is
a 2-bit
sat.
The table keeps track of the common-case counter
outcome for the branch
40
