Chapter - 4

Memory Systems and

Organization
Memory Challenges
 Ideally one desires a huge amount of very fast memory for
little cost, but:
 Fast memory is expensive
 Cheap memory is slow

 The solution on a fixed budget for memory is a hierarchy

 A small amount of very fast memory (Think SRAM)
 A medium amount of slower memory (Think DRAM)
 A large amount of slower yet memory (Think Disk)

 Comparing:
Technology Access Time Cost/GB
SRAM 0.5 – 5 ns $4,000
- $10,000
DRAM 50 – 70 ns $100 - $200
Disk 5 – 20 ms $0.50 - $2
 Typical Levels in a Hierarchical Memory
Capacity Access latency Cost per GB
100s B ns
Reg’s $Millions
10s KB a few ns
Cache 1
$100s Ks
MBs 10s ns
Cache 2 $10s Ks
100s MB 100s ns Speed
Main $1000s
gap
10s GB 10s ms Secondary $10s

TBs min+ Tertiary $1s

Fig. Names and key characteristics of levels in a memory hierarchy.

Computer Architecture, Memory System Design Slide 3

 Desktop, Drawer, and File Cabinet Analogy
Once the “working set” is in
the drawer, very few trips to
the file cabinet are needed.
Access cabinet
Access drawer
in 30 s
in 5 s

Register Access
file desktop in 2 s Cache
memory

Main
memory

Items on a desktop (register) or in a drawer (cache) are more

readily accessible than those in a file cabinet (main memory).

Computer Architecture, Memory System Design Slide 4

The Memory Hierarchy
 Take advantage of the principle of locality to
present the user with as much memory as is
available in the cheapest technology at the speed
offered by the fastest technology Inclusive–
Processor
what is in
4-8 bytes (words)
L1$ is a
subset of
Increasin L1$ what is in
g 8-32 bytes (block) L2$ is a
distance subset of
L2$
from the what is in
processor 1 to 4 blocks MM that is
in access Main Memory a subset of
time is in SM
1,024+ bytes (disk sector = page)

Secondary Memory

(Relative) size of the memory at each level

The Principal of Locality
 How does one UTILIZE the very fast memory effectively?

 Think “The Principal of Locality”

 Temporal Locality (Close in Time)
- Memory that has been accessed recently is most likely to be
accessed sooner
 Spatial Locality (Close in location)
- Memory that is close to memory that has been accessed recently is
most likely to be accessed sooner

 Organize memory in blocks

 Keep blocks likely to be used soon in the very fast memory
 Keep the next most likely blocks in medium fast memory
 Keep those not likely to be used soon in slower memory
 The Memory Hierarchy:
 Terminology
Hit: data is in some block in the upper level (Blk X)
 Hit Rate(h): the fraction of memory accesses found in the upper
level
 h = # of times word is found in cache/# of memory references/accesses
 Hit Time: Time to access the upper level which consists of
RAM access time + Time to determine hit/miss Lower Level
To Processor Upper Level Memory
Memory
Blk X
From Processor Blk Y

 Miss: data is not in the upper level so needs to be retrieve from

a block in the lower level (Blk Y)
 Miss Rate = 1 - (Hit Rate)
 Miss Penalty: Time to replace a block in the upper level
+ Time to deliver the block the
processor
General Organization of a Cache
Cache is an 1 valid t tag bits B = 2b bytes
array bit per per line per cache block
of sets line
valid tag 0 1 • • • B–1
Each set E lines
set 0: •••
contains per set
valid tag 0 1 • • • B–1
one or more
lines
valid tag 0 1 • • • B–1
Each line holds a set 1: •••
S = 2s sets
block of data valid tag 0 1 • • • B–1

•••
valid tag 0 1 • • • B–1
set S-1: •••
valid tag 0 1 • • • B–1

Cache size: C = B x E x S data bytes

8 15-213, S’08
General Organization of a Cache
Cache is an 1 valid t tag bits B = 2b bytes
array bit per per line per cache block
of sets line
valid tag 0 1 • • • B–1
Each set E lines
set 0: •••
contains per set
valid tag 0 1 • • • B–1
one or more
lines
valid tag 0 1 • • • B–1
Each line holds a set 1: •••
S = 2s sets
block of data valid tag 0 1 • • • B–1

•••
valid tag 0 1 • • • B–1
set S-1: •••
valid tag 0 1 • • • B–1

Cache size: C = B x E x S data bytes

9 15-213, S’08
 Addressing Caches
Address A:
t bits s bits b bits
m-1 0
v tag 0 1 • • •B–1
set 0: •••
v tag 0 1 • • •B–1 <tag> <set index><block offset>
v tag 0 1 • • •B–1
set 1: •••
v tag 0 1 • • •B–1

•••
The word at address A is in the cache if
v tag 0 1 • • •B–1
set S-1: ••• the tag bits in one of the <valid> lines in
v tag 0 1 • • •B–1 set <set index> match <tag>

The word contents begin at offset

<block offset> bytes from the beginning
of the block
10 15-213, S’08
 Addressing Caches
Address A:
t bits s bits b bits
m-1 0
v tag 0 1 • • •B–1
set 0: •••
v tag 0 1 • • •B–1 <tag> <set index><block offset>
v tag 0 1 • • •B–1
set 1: •••
v tag 0 1 • • •B–1

•••
1. Locate the set based on
v tag 0 1 • • •B–1
set S-1: ••• <set index>
v tag 0 1 • • •B–1 2. Locate the line in the set based on
<tag>
3. Check that the line is valid
4. Locate the data in the line based on
<block offset>
11 15-213, S’08
CACHE MEMORY MAPPING
There are three commonly used methods to
translate main memory addresses to cache
memory addresses.
 Direct-Mapped Cache
 Associative Mapped Cache
 Set-Associative Mapped Cache

The choice of cache mapping scheme affects

cost and performance, and there is no single
best method that is appropriate for all
situations
 Direct Mapped Block Placement

Cache

*0 *4 *8 *C address maps to block:

location = (block address MOD # blocks in cache)

00 04 08 0C 10 14 18 1C 20 24 28 2C 30 34 38 3C 40 44 48 4C

Memory

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 1: Mapping: 0 modulo 4 = 0

Mem Block DM Hit/Miss

Block 0
0
Block 1

Block 2

Block 3

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Direct-Mapped Cache
DM cache contains 4 1-word blocks. Find the # Misses for each cache
given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 1: Mapping: 0 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[0]

0 miss
Block 1

Block 2

Block 3

Set 0 is empty: write Mem[0]

CSCE430/830 Memory: Set-Associative $
 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 2: Mapping: 8 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[0]

0 miss
Block 1
8
Block 2

Block 3

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 2: Mapping: 8 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[8]

0 miss
Block 1
8 miss
Block 2

Block 3

CSCE430/830
Set 0 contains Mem[0]. Overwrite with Mem[8]
Memory: Set-Associative $
 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 3: Mapping: 0 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[8]

0 miss
Block 1
8 miss
0 Block 2

Block 3

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 3: Mapping: 0 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[0]

0 miss
Block 1
8 miss
0 miss Block 2

Block 3

CSCE430/830
Set 0 contains Mem[8]. Overwrite with Mem[0]
Memory: Set-Associative $
 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 4: Mapping: 6 mod 4 = 2

Mem Block DM Hit/Miss Block 0 Mem[0]

0 miss
Block 1
8 miss
0 miss Block 2
6
Block 3

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 4: Mapping: 6 mod 4 = 2

Mem Block DM Hit/Miss Block 0 Mem[0]

0 miss
Block 1
8 miss
0 miss Block 2 Mem[6]
6 miss
Block 3

Set 2 empty. Write Mem[6]

CSCE430/830 Memory: Set-Associative $
 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 5: Mapping: 8 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[0]

0 miss
Block 1
8 miss
0 miss Block 2 Mem[6]
6 miss
Block 3
8

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Direct-Mapped Cache
• DM cache contains 4 1-word blocks. Find the # Misses for each
cache given this sequence of memory block accesses: 0, 8, 0, 6, 8

DM Memory Access 5: Mapping: 8 mod 4 = 0

Mem Block DM Hit/Miss Block 0 Mem[8]

0 miss
Block 1
8 miss
0 miss Block 2 Mem[6]
6 miss
Block 3
8 miss

Set 0 contains Mem[0]. Overwrite with Mem[8]

CSCE430/830 Memory: Set-Associative $
 Direct-Mapped Cache with n one-word blocks
• Pros: find data fast
• Con: What if access 00001 and 10001 repeatedly?
 We always miss…

Cache

000
001

010

011

111
100
101

110

00001 00101 01001 01101 10001 10101 11001 11101

Memory
CSCE430/830 Memory: Set-Associative $
Example: Direct-Mapped Cache
Simplest kind of cache, easy to build
(only 1 tag compare required per access)

set 0: valid tag cache block E=1 lines per set

set 1: valid tag cache block

•••

set S-1: valid tag cache block

Cache size: C = B x S data bytes

25 15-213, S’08
Accessing Direct-Mapped Caches
Set selection

Use the set index bits to determine the set of interest.

set 0: valid tag cache block

selected set
set 1: valid tag cache block
•••

set S-1: valid tag cache block

t bits s bits b bits

00001
m-1 0
tag set index block offset
26 15-213, S’08
 Accessing Direct-Mapped Caches
Line matching and word selection

Line matching: Find a valid line in the selected set with a
matching tag

Word selection: Then extract the word

=1?(1) The valid bit must be set

0 1 2 3 4 5 6 7

selected set (i): 1 0110 b0 b1 b2 b3

(2) The tag bits in the
cache line must =? If (1) and (2), then cache hit
match the tag bits
in the address
t bits s bits b bits
0110 i 100
m-1
tag set index block offset0
27 15-213, S’08
Direct Mapped Cache Example
 One word/block, cache size = 1K words (4Kbyte)
Byte
31 30 ... 13 12 11 ... 2 1 0
offset

Tag 20 10 Data
Hit
Index
Index Valid Tag Data
0
1
2
.
.
.
1021
1022
1023
20 32

What kind of locality are we taking advantage of?

 Fully Associative Block Placement

Cache
arbitrary block mapping
location = any

00 04 08 0C 10 14 18 1C 20 24 28 2C 30 34 38 3C 40 44 48 4C

Memory

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 1:

Mem Block DM Hit/Miss S

0 e
t
0

FA Block Replacement Rule: replace least recently used block

CSCE430/830 Memory:in set
Set-Associative $
 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 1:

Mem Block DM Hit/Miss S Mem

0 miss e [0]
t
0

CSCE430/830 Set 0 is empty: write Mem[0] to Block

Memory: 0
Set-Associative $
 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 2:

Mem Block DM Hit/Miss S Mem

0 miss e [0]
8 t
0

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 2:

Mem Block DM Hit/Miss S Mem Mem

0 miss e [0] [8]
8 miss t
0

CSCE430/830 Blocks 1-3 are LRU: write Mem[8] to Block 1 Set-Associative $

Memory:
 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 3:

Mem Block DM Hit/Miss S Mem Mem

0 miss e [0] [8]
8 miss t
0
0

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 3:

Mem Block DM Hit/Miss S Mem Mem

0 miss e [0] [8]
8 miss t
0
0 hit

Block 0 contains Mem[0]

CSCE430/830 Memory: Set-Associative $
 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 4:

Mem Block DM Hit/Miss S Mem Mem

0 miss e [0] [8]
8 miss t
0 hit 0
6

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 4:

Mem Block DM Hit/Miss S Mem Mem Mem

0 miss e [0] [8] [6]
8 miss t
0
0 hit
6 miss

CSCE430/830 Blocks 2-3 are LRU : write Mem[6] toMemory:

Block 2
Set-Associative $
 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 5:

Mem Block DM Hit/Miss S Mem Mem Mem

0 miss e [0] [8] [6]
8 miss t
0 hit 0
6 miss
8

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Fully-Associative Cache
• Fully-Associative cache contains 4 1-word blocks. Find the # Misses
for each cache given this sequence of memory block accesses: 0, 8,
0, 6, 8

FA Memory Access 5:

Mem Block DM Hit/Miss S Mem Mem Mem

0 miss e [0] [8] [6]
8 miss t
0
0 hit
6 miss
8 hit

CSCE430/830 Block 1 contains Mem[8]Memory: Set-Associative $

 Fully-Associative Cache Basics
1 set, n blocks: no mapping restrictions on how blocks are
stored in cache: many ways, e.g. least recently used is
replaced (LRU)
0…0000
0…0001
0…0010

Example: 1-set, 8-block FA cache 0…0011

0…0100
0…0101
0…0110
0…0111
Set 0 Bloc Bloc Bloc Bloc Bloc Bloc Bloc Bloc
k0 k1 k2 k3 k4 k5 k6 k7 0…1000
0…1001
0…1010
0…1011
0…1100
0…1101
PRO: Less likely to replace needed data 0…1110
0…1111

CON: Must search entire cache for hit/miss …

CSCE430/830 Memory: Set-Associative $

 Set-Associative Block Placement

Cache

address maps to set:

*0 *0 *4 *4 *8 *8 *C *C location = (block address MOD # sets in cache)
(arbitrary location in set)

Set 0 Set 1 Set 2 Set 3

00 04 08 0C 10 14 18 1C 20 24 28 2C 30 34 38 3C 40 44 48 4C

Memory

CSCE430/830 Memory: Set-Associative $

 Set-Associative Cache Basics
Blocks are mapped from memory location to a specific set
in cache

0…0000 Mem block 0

0…0001
0…0010
0…0011
0…0100
0…0101
0…0110
Example: 4 set, 2- 0…0111
Set Block 0 Block 1
way SA cache 00 0…1000
0…1001
Mem block 8

(ADD mod 4) Set

01
0…1010
0…1011
Set 0…1100
10
0…1101

Set 0…1110
11 0…1111
…

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 1: Mapping: 0 mod 2 = 0

Mem Block DM Hit/Miss Set 0

0
Set 1

SA Block Replacement Rule: replace least recently used block

CSCE430/830 Memory:in set
Set-Associative $
 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 1: Mapping: 0 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0]

0 miss
Set 1

CSCE430/830 Set 0 is empty: write Mem[0] to Block

Memory: 0
Set-Associative $
 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 2: Mapping: 8 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0]

0 miss
Set 1
8

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 2: Mapping: 8 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0] Mem[8]

0 miss
Set 1
8 miss

CSCE430/830 Set 0, Block 1 is LRU: write Mem[8]

Memory: Set-Associative $
 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 3: Mapping: 0 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0] Mem[8]

0 miss
Set 1
8 miss
0

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 3: Mapping: 0 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0] Mem[8]

0 miss
Set 1
8 miss
0 hit

CSCE430/830 Set 0, Block 0 contains Mem[0]Memory: Set-Associative $

 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 4: Mapping: 6 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0] Mem[8]

0 miss
Set 1
8 miss
0 hit
6

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 4: Mapping: 6 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0] Mem[6]

0 miss
Set 1
8 miss
0 hit
6 miss

CSCE430/830 Set 0, Block 1 is LRU: overwrite withMemory:

Mem[6]
Set-Associative $
 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 5: Mapping: 8 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[0] Mem[6]

0 miss
Set 1
8 miss
0 hit
6 miss
8

CSCE430/830 Memory: Set-Associative $

 Example: Accessing A Set-Associative Cache
• 2-way Set-Associative cache contains 2 sets, 2 one-word blocks each.
Find the # Misses for each cache given this sequence of memory block
accesses: 0, 8, 0, 6, 8

SA Memory Access 5: Mapping: 8 mod 2 = 0

Mem Block DM Hit/Miss Set 0 Mem[8] Mem[6]

0 miss
Set 1
8 miss
0 hit
6 miss
8 miss

CSCE430/830 Set 0, Block 0 is LRU: overwrite withMemory:

Mem[8]
Set-Associative $
 Set-Associative Cache Basics
blocks are mapped from memory location to a specific
set in cache

0…0000 Mem block 0

0…0001
0…0010
0…0011
0…0100
0…0101

Example: 4 set, 2-way SA cache 0…0110

0…0111
(X mod 4) 0…1000 Mem block 8

PRO: Set
00
Block 0 Block 1
0…1001
0…1010

Easier to find but won’t Set

01
0…1011
0…1100
always overwrite Set
10
0…1101
0…1110

CON: Set
11 0…1111
…
Must search set for hit/miss
CSCE430/830 Memory: Set-Associative $
 Set Associative Cache Design

Address
• Key idea: 31 30 12 11 10 9 8 3210
– Divide cache into sets
– Allow block anywhere in a set
22 8
• Advantages:
– Better hit rate
• Disadvantage:
– More tag bits Index V Tag Data V Tag Data V Tag Data V Tag Data
– More hardware 0
– Higher access time 1
2

253
254
255
22 32

4-to-1 multiplexor

Hit Data

A Four-Way Set-Associative Cache

CSCE430/830 Memory: Set-Associative $