Lecture 9: Dynamic Scheduling
Kunle Olukotun
Gates 302
[email protected]
http://www-leland.stanford.edu/class/ee282h/
1
Hardware schemes for ILP
Why HW at runtime?
works when compiler cant know dependencies or latencies (e.g.
cache miss)
compiler is simpler
code is less pipeline dependent
In our pipeline so far, if an instruction stalls, everything after it
stalls too (in-order execution):
divd
addd
subd
fp0, fp2, fp4
fp10, fp0, fp8
fp8, fp8, fp14
;this takes a long time
;and this must stall for it
;but why couldnt this go ahead?
We want to be able to decode instructions into the pipe in order
but then let them stall individually while waiting for operands
before issue to execution units.
Dynamic scheduling (out-of-order issue/execution)
Dynamic Scheduling
Basic Concept
Sequential
Instruction Stream
PC
LW
LW
ADD
SW
LW
LW
ADD
SW
LW
LW
ADD
SW
LW
LW
R1,A
R2,B
R3,R1,R2
R3,C
R4,8(A)
R5,8(B)
R6,R4,R5
R6,8(C)
R7,16(A)
R8,16(B)
R9,R7,R8
R9,16(C)
R10,24(A)
R11,24(B)
Window of
Waiting Instructions
on operands & resources
ADD
SW
ADD
SW
LW
LW
ADD
SW
LW
LW
R3,R1,R2
R3,C
R6,R4,R5
R6,8(C)
R7,16(A)
R8,16(B)
R9,R7,R8
R9,16(C)
R10,24(A)
R11,24(B)
Execution
Resources
Register File
Issue Logic
Scoreboarding:
The big picture
Functional unit
Register
File
Functional unit
Functional unit
Functional unit
Scoreboard
All hazard detection
and scheduling is
centralized in the
scoreboard.
The pipeline now has an extra stage (even ignoring load/stores), plus
possible wait cycles for clearance from the scoreboard
IF: instruction fetch
ID: instruction decode. Stall the pipe if structural or WAW hazard.
SB: scoreboard: wait for any RAW hazards to clear
RF: read operands from the register file
EX: execute in a functional unit,
WB: wait for any WAR hazard to clear, then write back to register file
4
Scoreboarding:
Pipelined example
multd
sd
addu
neg
multd
sd
addu
neg
1
IF
2
ID
IF
fp0, fp2, fp4
fp0, 0(r4)
r4, r2, #8
fp0, fp4
3
SB
ID
IF
4
RF
SB
ID
IF
;4 cycles
;2 cycles
;2 cycles
;1 cycle
5
EX
SB
SB
ID
6
EX
SB
RF
ID
7
EX
SB
EX
ID
8
9
10 11 12
EX WB
SB SB RF EX WB
EX --- WB
ID ID SB RF EX WB
Scoreboarding:
The data structures
Instruction Instruction
status
Issued?
Operands
read?
Execution
complete?
Result
written?
FU status
Functional
unit
Busy?
Op
Dest
reg
reg
Source 1
from FU ready?
reg
Source 2
from FU ready?
Result registers pending
FP0
FP2
FP4
FP6
FP8
FP10
... FP30
Result will come
from which FU?
6
Scoreboarding:
The rules
Issue
IF: FU not busy (no structural hazards) and no result pending for
destination register (no WAW hazards)
THEN: Assign FU entry.
From FU comes from entry in register result pending
Ready? (which means ready but not yet read) is TRUE only
if there is no register result pending for that register
Register file
IF: Source1 and Source2 are both ready (no RAW hazards)
THEN: ready=FALSE for both, read registers, and start EX
Writeback
IF: None of the functional units needs to use our result register and
has yet to read it (no WAR hazards)
THEN: Make ready any functional unit sources waiting for this FU,
write the register file, clear the FU status, and clear our
destinations register result pending entry.
7
Scoreboarding:
A detailed example
Instruction
status
FU status
Instruction
ld f6, ...
ld f2, ...
multd f0,f2,f4
subd f8, f6, f2
divd f10, f0, f6
addd f6, f8, f2
Issued?
y
y
y
y
y
Operands
read?
y
y
Execution
complete?
y
y
Functional
unit
Busy?
Op
Dest
reg
Integer
Mult 1
Mult 2
Add
Divide
y
y
n
y
y
load
mult
f2
f0
f3
f2
integer
n
n
-f4
sub
div
f8
f10
f6
f0
Mult 1
y
n
f2
f6
FP4 FP6
FP8
FP10
Add
Divide
reg
Source 1
from FU ready?
reg
Result
written?
y
Source 2
from FU ready?
y
Integer
n
y
Result registers pending
Result will come
from which FU?
FP0
FP2
Mult 1
Integer
...
FP30
Scoreboarding:
Summary
+ Permits out-of-order execution and out-of-order completion for nondata-dependent operations
+ Centralizes decision making -- about as big as one functional unit
in the 6600
- Requires lots of busses to/from register file and functional units
- No forwarding: all operands come from the register file
But since results are written as soon as possible (without waiting through
MEM, etc.), the benefit of adding forwarding is only one cycle
- Stalls for all WAR and WAW hazards
Reducing WAR and WAR hazards:
Compiler (static) register renaming
Consider the fragment
divd
fp1, fp2, fp3
subd fp2, fp3, fp4
;this will take a long time
;this cant start until divd is finished
;with fp2
std
fp2, 0(r1)
There is a WAR antidependence on fp2 between the divd and the
subd.
But by renaming fp2 and subsequent uses of it we avoid the
hazard:
divd
fp1, fp2, fp3
;this will take a long time
subd fp5, fp3, fp4
;but now this can start anytime
std
fp5, 0(r1)
Reduces stalls for some pipelines, but uses additional registers
10
Register renaming again:
this time in hardware!
The trick: Every time you write to a register number, allocate a new
physical register. Think of it as multiple copies of each register, one
for each new value written to it.
Implementation implications:
Need more physical registers than register addresses
Keep a pool of unassigned physical registers, and allocate one when a
register is written
All uses to the original register number must be remapped to the new
physical register
When a register is written, all previous physical registers allocated for it
can be freed as soon as all previous instructions which use it have
completed
The big win over scoreboarding: no stalls for WAR and WAW
hazards!
11
Dynamic register renaming:
an example
Original code
multd fp0, fp2, fp4
sd
fp0, 0(r4)
addu r4, r2, #8
addd fp0, fp6, fp8
Two WAR hazards: r4 (sd/addu), and fp0 (sd/addd)
Renamed code
multd fp0a, fp2a, fp4a
sd
fp0a, 0(r4a)
addu r4b, r2a, #8
addd fp0b, fp6a, fp8a
12
Register Renaming
Use additional physical registers
ADD
R23
R34
R37
R16
R22
R1
R2
Fetch:
Assign free register to dest
Rename sources with map
Issue:
When sources available
Fetch sources from reg file
R3
R0:R0
R1:R18
R2:R47
R3:R19
Working Execute: Write result to reg file
Map
Free List
ADD
R1
R23 R47 0 R19 1
R0
R31
R127
13
Tomasulos Algorithm:
The big picture
Replace the functional unit status table with reservation
stations (distributed window):
Each function unit has a set of reservations stations for instructions
waiting to use it
The reservation station stores operand values as they become
available, or the name of a reservation station that will provide the
result. Thus the original register number is no longer relevant, and
is now renamed as a reservation station
Each result is broadcast to the Common Data Bus (CDB) and
can be snagged by any reservation station that needs it.
Instructions can now be issued even if there are structural
hazards or WAR/WAW hazards.
14
Tomasulos Algorithm:
Hardware Structure
Comon data bus
Memory
Tags
Register
file
1
2
3
load 7
buffers 8
9
4
5
6
Functional
unit
Functional
unit
store
buffers
Res
e
stati rvation
ons
15
Tomasulos Algorithm:
The pipeline stages
Pipeline stages
IF: instruction fetch
ID: instruction decode
reservation station assignment (may stall)
operand fetch
RS: reservation station, until RAW hazards clear
EX: execute, arbitrate for CDB
WB: write results to CDB (and register file)
How data hazards are eliminated:
RAW: delay in reservation station until operands ready
WAR: fetch operands early and in issue order
WAW: only last writer (in issue order) will update the register file
16
Tomasulos algorithm:
Pipeline example
multd
sd
addu
ld
multd
sd
addu
ld
1
IF
2
ID
IF
fp0, fp2, fp4
fp0, 0(r4)
r4, r2, #8
fp0, 0(r5)
;4 cycles
;2 cycles
;2 cycles
;2 cycles
3
RS
ID
IF
6
EX
RS
EX
RS
4
EX
RS
ID
IF
5
EX
RS
RS
ID
7
EX
RS
EX
EX
8
WB
RS
-EX
10
11
EX
WB
--
EX
WB
12
WB
CDB stalls
17
Tomasulos Algorithm:
The data structures
Reservation stations
Functional
unit
Busy?
Source 1
value
tag
Op
Source 2
value
tag
Add1
Add2
Add3
Mult1
Mult2
xx
store buffers
busy? tag
Reservation stn #
or load buffer #
load buffers
addr/value
busy?
addr
1
2
3
Five functional units with
one reservation stn each.
Register status
FP0
FP2
FP4
FP6
FP8
FP10
... FP30
Result will come
from which resvn stn
18
Tomasulos Algorithm:
The rules
Issue
IF: a reservation station is available
THEN: Assign a reservation station entry
If a source register has no resulting pending. then read its value from the
file; otherwise set the tag to the station from which the result will come.
Set the register status for our result register to our station # (Note that it
may override something already there!)
Execute
IF: source1 and source2 values are present
THEN: begin execution
Writeback
IF: CDB slot is available
THEN:
Write to all registers for which were the destination, and zero the register
status.
In all reservation stations or store buffers that list us as an operand tag,
write the value and clear the tag.
19
Tomasulo and Memory Operations
Maintain ordering with respect to stores
A load or store cannot execute before (after) a
preceding (following) store unless they are
known to be to different addresses
Memory disambiguation
hard at compile time, easy at run time
20
Tomasulos Algorithm:
An example
Reservation stations
Functional
unit
Busy?
Op
Add1
Add2
Add3
Mult1
Mult2
y
y
n
y
y
sub
add
Source 2
value
tag
xxxxxxxx
Load2
Load2
Add1
mul
div
store buffers
busy? tag
Source 1
value
tag
Load2
Mult1
load buffers
addr/value
1
2
3
busy?
n
y
n
addr
xxx
yyyyyyyy
zzzzzzzz
Instruction Issued? Exec? WB?
ld f6, xxx
y
y
y
ld f2, xxx
y
y
multd f0, f2, f4
y
subd f8, f6, f2
y
divd f10, f0, f6
y
addd f6, f8, f2
y
Register status
Result will come from
which resvn stn
FP0
FP2
Mult1
Load2
FP4
FP6
FP8
FP10
Add2
Add1
Mult2
...
FP30
21
Tomasulos Algorithm:
A big winner in loops
If we can handle branches well, Tomasulos algorithm allows parallel
execution of multiple iterations of a loop body just like compiler loop
unrolling.
; for (i=0; i<100; ++i) a[i] = a[i] * c;
loop:
ld
f0, 0(r1)
multd
f4, f0, f2
sd
0(r1), f4
subi
r1, r1, #8
bnez
r1, loop
With two multd units, two iterations of this loop will issue and execute in
parallel.
22
Tomasulos Algorithm:
Notes and Summary
Still need a contention strategy for:
Access to the CDB
Access to functional units with more than one reservation station
Combination of FIFO and priority?
Need to check for memory hazards: loads must check the
addresses in the store buffers (dynamic memory
disambiguation)
Advantages
Increases parallelism by dynamic scheduling
Eliminates WAR and WAW hazards by dynamic register renaming
Expands the available registers without affecting programs or
compilers
Disadvantages:
Lots of associative (parallel) comparisons
Imprecise interrupts (need reorder buffer or equivalent)
Branch hazards are still a problem
23
Extracting more parallelism:
Speculation
ILP is difficult to find in some common loops :
for (p=head; p!=NIL; p = p->link) ++p->value;
loop:
test:
j
lw
addi
sw
lw
bnez
test
r5, 0(r4)
r5, r5, #1
r5, 0(r4)
r4, 4(r4)
r4, loop
So far, we have predicted conditional branches and fetched based
on the prediction. Now: actually execute the predicted instructions
conditionally, and be prepared to nullify them.
Speculative execution works well combined with dynamic scheduling,
especially Tomasulos algorithm
Issue in-order, execute out-of-order, commit in-order
Combining reservation stations with a reorder buffer gives us speculative
execution and precise interrupts!
24
Tomasulos algorithm
with speculation
to memory
Reorder
buffer
also
re
load places
and s
tore b
uf
Comon data bus
Register
file
fers
1
2
3
Tags
4
5
6
Functional
unit
Res
e
stati rvation
ons
Functional
unit
25
Tomasulos Algorithm with
speculation: The data structures
Reservation stations
Functional
unit
Busy?
Op
Dest
Add1
Add2
Add3
Mult1
Mult2
n
n
n
n
y
mult
div
#3
#5
entry
1
2
3
4
5
6
busy?
n
n
n
y
y
y
Reorder
buffer
#3
Source 2
value
tag
yyyyyyyy
ssssssss
xxxxxxxx
#3
Instr
ld f6, 34(r2)
ld f6, 45(r3)
multd f0, f2, f4
subd f8, f6,f2
divd f10, f0, f6
addd f6, f8, f2
FP0
Register
Result will be in
status which reorder buffer?
Source 1
value
tag
FP2
FP4
State
commit
commit
writeback
writeback
execute
writeback
dest
f6
f2
f0
f8
f10
f6
value
xxxxxxxx
yyyyyyyy
zzzzzzzz
wwwwww
uuuuuuuu
FP6
FP8
FP10
#6
#4
#5
... FP30
26
Tomasulos Algorithm with
speculation: The rules
Issue
IF: reservation station and reorder buffer slot is available
THEN: queue for execution
fill in the reservation station entry (copying operands from reorder buffer or
register file)
fill in the reorder buffer entry, state = execute
Execute
WHEN source1 and source2 values are present or broadcast on CDB
THEN begin execution
Writeback
WHEN: CDB slot is available
THEN: Free the reservation station
Write to all reorder buffer slots for which were the destination
In all reservation stations that list our buffer slot as an operand tag, write the
value and clear the tag
Commit
WHEN we reach the top of the reorder buffer and have a value
THEN:
Write the register (or store into memory)
Zero the register status and free the reorder buffer entry
27
Branches and Exceptions
When a branch occurs
Make a reorder buffer entry for branch
Predict the direction and fetch/execute
When the branch commits (reaches the top of the reorder buffer)
If the prediction was wrong, clear the reorder buffer and start fetching
from the other path
Works great for correct prediction, but misprediction penalty is high
Might partially clear the reorder buffer and restart fetch as soon as the
branch condition is resolved, rather than waiting for it to be at the top
Even more exotic: speculatively execute both directions! (Many
machines fetch both directions, but few execute.)
Exceptions:
Record exceptions in the reorder buffer, and dont recognize it until its at
the top. Interrupts are precise, and the register status array takes the
place of the future file.
28
