Computer Architecture

Assoc. Prof. Nguyễn Trí Thành, PhD

UNIVERSITY OF ENGINEERING AND TECHNOLOGY
FACULTY OF INFORMATION TECHNOLOGY
DEPARTMENT OF INFORMATION SYSTEMS
[email protected]
Chapter 4: Cache

2
 Structural Hazards - Reminded
 Structural hazard: inadequate hardware to simultaneously support
all instructions in the pipeline in the same clock cycle
 E.g., suppose single – not separate – instruction and data memory
in pipeline below with one read port
 then a structural hazard between first and fourth lw instructions

Program
execution 2 4 6 8 10 12 14
Time
order
(in instructions)
Instruction Data
lw $1, 100($0) Reg ALU Reg
fetch access
Pipelined
Instruction Data
lw $2, 200($0) 2 ns Reg ALU Reg
fetch access
Hazard if single memory
Instruction Data
lw $3, 300($0) 2 ns Reg ALU Reg
fetch access
Instruction Data
lw $4, 400($0) Reg ALU Reg
2 ns fetch access

2 ns 2 ns 2 ns 2 ns 2 ns

 MIPS was designed to be pipelined: structural hazards are easy to

avoid!
4
5
6
7
8
 Memory Hierarchy Levels
Capacity
Access Time Staging
Cost Xfer Unit
CPU Registers
Registers Upper Level
100s Bytes
300 – 500 ps (0.3-0.5 ns) prog./compiler
Instr. Operands 1-8 bytes faster
L1 and L2 Cache L1 Cache
10s-100s K Bytes cache cntl
~1 ns - ~10 ns Blocks 32-64 bytes
$1000s/ GByte
L2 Cache
cache cntl
Main Memory Blocks 64-128 bytes
G Bytes
80ns- 200ns Memory
~ $100/ GByte
OS
Pages 4K-8K bytes
Disk
10s T Bytes, 10 ms
(10,000,000 ns)
Disk
~ $1 / GByte user/operator
Files Mbytes
Larger
Tape
infinite Tape Lower Level
sec-min 9
~$1 / GByte
Principle of Locality
 Programs access a small proportion of their
address space at any time
 Temporal locality
 Items accessed recently are likely to be accessed
again soon
 e.g., instructions in a loop, induction variables
 Spatial locality
 Items near those accessed recently are likely to
be accessed soon
 E.g., sequential instruction access, array data
10
Control structure

• Loop

 Sequential

lw $t2, 0($t3)
lw $t3, 4($t3)
beq $t2, $t3, Label #assume not equal
add $t5, $t2, $t3
sw $t5, 8($t3)
Label: j 32
11
Taking Advantage of Locality

 Memory hierarchy
 Store everything on disk
 Copy recently accessed (and nearby) items
from disk to smaller DRAM memory
 Main memory
 Copy more recently accessed (and nearby)
items from DRAM to smaller SRAM memory
 Cache memory attached to CPU

12
Cache position

13
Intel

14
Memory Hierarchy Levels
 Block (aka line): unit of copying
 May be multiple words
 If accessed data is present in
upper level
 Hit: access satisfied by upper level
 Hit ratio: hits/accesses
 If accessed data is absent
 Miss: block copied from lower level
 Time taken: miss penalty
 Miss ratio: misses/accesses
= 1 – hit ratio
 Then accessed data supplied from
upper level
Direct Mapped Cache
 Location determined by address
 Direct mapped: only one choice
 (Block address) modulo (#Blocks in cache)

 #Blocks is a
power of 2
 Use low-order
address bits

16
 §5.2 The Basics of Caches
Cache Memory
 Cache memory
 The level of the memory hierarchy closest to the
CPU
 Given accesses X1, …, Xn–1, Xn

 How do we know if
the data is present?
 Where do we look?

17
Tags and Valid Bits

 How do we know which particular block is

stored in a cache location?
 Store block address as well as the data
 Actually, only need the high-order bits
 Called the tag
 What if there is no data in a location?
 Valid bit: 1 = present, 0 = not present
 Initially 0

18
Cache Example
 8-blocks, 1 word/block, direct mapped
 Initial state

Index V Tag Data

000 N
001 N
010 N
011 N
100 N
101 N
110 N
111 N
19
Cache Example (cont’d)
Word addr Binary addr Hit/miss Cache block
22 10 110 Miss 110

Index V Tag Data

000 N
001 N
010 N
011 N
100 N
101 N
110 Y 10 Mem[10110]
111 N
20
Cache Example (cont’d)
Word addr Binary addr Hit/miss Cache block
26 11 010 Miss 010

Index V Tag Data

000 N
001 N
010 Y 11 Mem[11010]
011 N
100 N
101 N
110 Y 10 Mem[10110]
111 N
21
Cache Example (cont’d)
Word addr Binary addr Hit/miss Cache block
22 10 110 Hit 110
26 11 010 Hit 010

Index V Tag Data

000 N
001 N
010 Y 11 Mem[11010]
011 N
100 N
101 N
110 Y 10 Mem[10110]
111 N
22
Cache Example (cont’d)
Word addr Binary addr Hit/miss Cache block
16 10 000 Miss 000
3 00 011 Miss 011
16 10 000 Hit 000

Index V Tag Data

000 Y 10 Mem[10000]
001 N
010 Y 11 Mem[11010]
011 Y 00 Mem[00011]
100 N
101 N
110 Y 10 Mem[10110]
111 N
23
Cache Example (cont’d)
Word addr Binary addr Hit/miss Cache block
18 10 010 Miss 010

Index V Tag Data

000 Y 10 Mem[10000]
001 N
010 Y 10 Mem[10010]
011 Y 00 Mem[00011]
100 N
101 N
110 Y 10 Mem[10110]
111 N
24
Mapping from memory to
cache

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 …
0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 …
Block offset Block offset Block offset Block offset …
0 1 2 3 …
Program is divided into equal chunks of the same size as the cache block
BlockIndex = x / block_size
BlockOffset = x mod block_size
For example: x=14
• BlockIndex = 14 / 4 = 3
• BlockOffset = 14 mod 4 = 2
Map: use BlockIndex for calculating the Block in the cache
Address Subdivision

26
Example: Larger Block Size
 64 blocks, 16 bytes/block, 32 bit address regs
 To what block number does address 1200 map?
 Offset=?
 Index=?
 Tag=?
31 10 9 4 3 0
Tag Index Offset
22 bits 6 bits 4 bits

27
Cache usage protocol

 Identify the index, tag, offset of the reference

 Check the valid bit
 If v=valid, check the tag
 If two tags are equal => return the data (cache hit)
 Else raise a cache miss => Load the data into the
WHOLE slot, update tag and v=valid, return the data
 If v=invalid, raise a cache miss
 Load the data into the WHOLE slot, update tag and
v=valid, return the data

28
Example
 Suppose a cache 16KB, slot size=32 bytes,
address reg=32 bit. Identify the bits for tag,
index and offset (ignore tag and valid bit).
 Given the above cache with direct map
 Cache with 64 slots, slot size=32 bytes, address
reg=16 bit
 Show hit/miss for the reference sequence of MIPS
instructions: 184, 188, 192, 196, 200, 204, 208,
212, 216, 192, 196, 200, 204.

29
Associative Caches
 Fully associative
 Allow a given block to go in any cache entry
 Requires all entries to be searched at once
 Comparator per entry (expensive)
 n-way set associative
 Each set contains n entries
 Block number determines which set
 (Block number) modulo (#Sets in cache)
 Search all entries in a given set at once
 n comparators (less expensive)
30
Associative Cache Example

31
Spectrum of Associativity
 For a cache with 8 entries

32
Associativity Example
 Compare 4-block caches
 Direct mapped, 2-way set associative,
fully associative
 Block access sequence: 0, 8, 0, 6, 8

 Direct mapped
Block Cache Hit/miss Cache content after access
address index 0 1 2 3
0 0 miss Mem[0]
8 0 miss Mem[8]
0 0 miss Mem[0]
6 2 miss Mem[0] Mem[6]
8 0 miss Mem[8] Mem[6]

33
Associativity Example
 2-way set associative
Block Cache Hit/miss Cache content after access
address index Set 0 Set 1
0 0 miss Mem[0]
8 0 miss Mem[0] Mem[8]
0 0 hit Mem[0] Mem[8]
6 0 miss Mem[0] Mem[6]
8 0 miss Mem[8] Mem[6]

 Fully associative
Block Hit/miss Cache content after access
address
0 miss Mem[0]
8 miss Mem[0] Mem[8]
0 hit Mem[0] Mem[8]
6 miss Mem[0] Mem[8] Mem[6]
8 hit Mem[0] Mem[8] Mem[6]

34
How Much Associativity

 Increased associativity decreases miss rate

 But with diminishing returns
 Simulation of a system with 64KB
D-cache, 16-word blocks, SPEC2000
 1-way: 10.3%
 2-way: 8.6%
 4-way: 8.3%
 8-way: 8.1%

35
Set Associative Cache
Organization

36
Exercise

 Given a 4-way cache size =128KB (for data),

block size=32 bytes, the address register=32
bit.
 Identify the components of the register.
 Given a reference x=1864, identify the
components

37
Replacement Policy

 Direct mapped: no choice

 Set associative
 Prefer non-valid entry, if there is one
 Otherwise, choose among entries in the set
 Least-recently used (LRU)
 Choose the one unused for the longest time
 Simple for 2-way, manageable for 4-way, too hard
beyond that
 Random
 Gives approximately the same performance as
LRU for high associativity
38
Block Size Considerations

 Larger blocks should reduce miss rate

 Due to spatial locality
 But in a fixed-sized cache
 Larger blocks  fewer of them
 More competition  increased miss rate
 Larger blocks  pollution
 Larger miss penalty
 Can override benefit of reduced miss rate
 Early restart and critical-word-first can help
39
Cache Misses

 On cache hit, CPU proceeds normally

 On cache miss
 Stall the CPU pipeline
 Fetch block from next level of hierarchy
 Instruction cache miss
 Restart instruction fetch
 Data cache miss
 Complete data access

40
Write-Through
 On data-write hit, could just update the block in
cache
 But then cache and memory would be inconsistent
 Write through: also update memory
 But makes writes take longer
 e.g., if base CPI = 1, 10% of instructions are stores,
write to memory takes 100 cycles
 Effective CPI = 1 + 0.1×100 = 11
 Solution: write buffer
 Holds data waiting to be written to memory
 CPU continues immediately
 Only stalls on write if write buffer is already full
41
Write-Back

 Alternative: On data-write hit, just update the

block in cache
 Keep track of whether each block is dirty
 When a dirty block is replaced
 Write it back to memory
 Can use a write buffer to allow replacing block to
be read first

42
Write Allocation

 What should happen on a write miss?

 Alternatives for write-through
 Allocate on miss: fetch the block
 Write around: don’t fetch the block
 Since programs often write a whole block before
reading it (e.g., initialization)
 For write-back
 Usually fetch the block

43
 Main Memory Supporting Caches
 Use DRAMs for main memory
 Fixed width (e.g., 1 word)
 Connected by fixed-width clocked bus
 Bus clock is typically slower than CPU clock
 Example cache block read
 1 bus cycle for address transfer
 15 bus cycles per DRAM access
 1 bus cycle per data transfer
 For 4-word block, 1-word-wide DRAM
 Miss penalty = 1 + 4×15 + 4×1 = 65 bus cycles
 Bandwidth = 16 bytes / 65 cycles = 0.25 B/cycle
44
Increasing Memory Bandwidth

 4-word wide memory

 Miss penalty = 1 + 15 + 1 = 17 bus cycles
 Bandwidth = 16 bytes / 17 cycles = 0.94 B/cycle
 4-bank interleaved memory
 Miss penalty = 1 + 15 + 4×1 = 20 bus cycles
 Bandwidth = 16 bytes / 20 cycles = 0.8 B/cycle
45
Average Access Time

 Hit time is also important for performance

 Average memory access time (AMAT)
 AMAT = Hit time + Miss rate × Miss penalty
 Effective CPI
 ECPI=CPI (ideal)+miss rate1 × Miss penalty + …
 Example
 CPU with 1ns clock, hit time = 1 cycle, miss
penalty = 20 cycles, I-cache miss rate = 5%
 AMAT = 1 + 0.05 × 20 = 2
 2 cycles per instruction 46
Cache Performance Example
 Given
 I-cache (instruction) miss rate = 2%
 D-cache (data) miss rate = 4%
 Miss penalty = 100 cycles
 Base CPI (ideal cache) = 2
 Load & stores are 36% of instructions
 Miss cycles per instruction
 I-cache: 0.02 × 100 = 2
 D-cache: 0.36 × 0.04 × 100 = 1.44
 Actual CPI = 2 + 2 + 1.44 = 5.44
 Ideal CPU is 5.44/2 =2.72 times faster
47
Multilevel Caches

 Primary cache attached to CPU

 Small, but fast
 Level-2 cache services misses from primary
cache
 Larger, slower, but still faster than main memory
 Main memory services L-2 cache misses
 Some high-end systems include L-3 cache

48
Multilevel Cache Example

 Given
 CPU base CPI = 1, clock rate = 4GHz
 Miss rate/instruction = 2%
 Main memory access time = 100ns
 With just primary cache
 CPU clock cycle=1/(4*10^9) = ¼ ns=0.25ns
 Miss penalty = 100ns/0.25ns = 400 cycles
 Effective CPI = 1 + 0.02 × 400 = 9

49
Example (cont.)
 Now add L-2 cache
 Access time = 5ns
 Global miss rate to main memory = 0.5%
 Primary miss with L-2 hit
 Penalty = 5ns/0.25ns = 20 cycles
 Primary miss with L-2 miss
 Extra penalty = 500 cycles
 CPI = 1 + 0.02 × 20 + 0.005 × 500 = 3.9
 Performance ratio = 9/3.9 = 2.3
50
Performance Summary

 When CPU performance increased

 Miss penalty becomes more significant
 Decreasing base CPI
 Greater proportion of time spent on memory stalls
 Increasing clock rate
 Memory stalls account for more CPU cycles
 Can’t neglect cache behavior when
evaluating system performance
51