1: Introduction

Cache memory is a key mechanism for improving overall system

performance. A cache exploits the locality inherent in the reference stream
of the typical application. Two primary type of locality are available and the
degrees to which they can be exploited depend on the program input
characteristic. Temporal locality relies on the greater probability that recently
accessed data will be accessed in near future. Spatial locality refers to the
tendency for adjacent or nearby locations to be reference together in time.

Prefetching mechanism can be used to reduce cache misses. It

also reduces cache stall time by bringing data into cache before it use, so that it
can be accessed without delay. Hardware-based prefetching requires some
modification to of the cache, but little modification to processor core. Its main
advantage is that prefetches are handled dynamically at run time without
compiler intervention. The drawbacks are that extra hardware is needed and the
memory reference for complex access pattern is difficult to predict. In contrast
software-based approaches rely on compiler technology to perform static
program analysis and to selectively insert prefetch instructions. The draw backs
are the nonnegligible execution overhead is introduced due to extra instructions
and that static analysis may misses some prefetch opportunities that are
obvious at runtime.

Most hardware prefetching mechanisms generate a

prefetch signal when either a cache miss or hit occurs. Therefore a large
number of prefetch signal are generated, leading to higher power
 consumption and cache pollution. Prefetching can also cause an increase in
the MCPI (Memory Cycle Per Instruction) in the worst case. This intelligent
prefetching mechanism avoids these problems by reducing the prefetch
generation rate. As a result, power consumption is also reduced and increase
in MCPI is negligible.

A SMI cache is constructed in three parts; a conventional direct

mapped cache With a small block size, a fully associative buffer with a large
block size at the same cache level, and a hardware prefetching unit. The
improvement in performance is achieved by exploiting the basic characteristic
of locality. Specifically, two different block size are used, i.e., a small block
size to exploit temporal locality and a large block size that is multiple of the
small block size to exploit spatial locality. Also, temporal locality is enhanced
by selectively retaining block with a high probability of repeated reference in
time domain, and spatial locality is enhanced by both a large fetch size and an
intelligent prefetching mechanism. The technique we used for enhancing
temporal locality is to monitor the behavior of a block over a time interval
before choosing to store it into the direct mapped cache. That is, instead of
placing every missed block directly into the direct mapped cache, we first load
a large block including the missed block into the spatial buffer. The missed
block is not moved into the direct mapped cache until the large block is
replaced from the spatial buffer, and then only if it has shown evidence of
temporal locality while it was resident in the spatial buffer. Thus the SMI cache
exploits information about utilization of the block that is obtained during this
time interval. The interval itself is the proportional of the entries in the spatial
buffer. The behavior that we observe during this time enables us to determine
which block show strong temporal locality.
 The average miss ratio and memory access time of the SMI
cache for a given cache space (e.g., 8 KB) are equivalent to those of a
conventional direct-mapped cache with four times as much space (e.g., 32 KB).
Thus, a 60 to 80 percent reduction in chip area can be obtained over a direct
mapped cache while achieving a high performance gain. Also, the SMI cache
can reduce the miss ratio by around 20% and the average memory access time
by 10%, compared with a victim cache.

2: Selective-Mode Intelligence Cache System

The motivation for the design, the architectural model for the SMI cache
system, and its operation in the context of hardware prefetching is as follows.

2.1: Motivation:

Common design objective for the cache are to improve utilization of

the temporal and spatial locality inherent in applications. However no single
cache organization exploit both temporal and spatial locality optimally because
of their contradictory characteristics.
Increasing block size reduces the number of cache blocks. Thus, conventional
cache design attempt to compromise by selecting an adequate block size. The
cache’s lack of adaptability to pattern of references of different types of locality
poses a further drawback. A fixed block size result in suboptimal overall access
time and bandwidth
Utilization because it fails to match the varying spatial and temporal locality
across and within programs. A cache system that provides two caches with
different configurations is an effective structure for overcoming these structural
drawbacks.
A significant reason for using a direct map catch with a small block size is to
reduce power consumption and cycle time. To make up for the poor spatial
locality of such a direct mapped cache, we provide a fully associative spatial
buffer with a large block size. The small block size exploits temporal locality
and the large block size exploits spatial locality. Small blocks are first loaded
as part of larger blocks that are fetched into the spatial buffer. We selectively
extend the lifetime of those small blocks that exhibits high temporal locality by
storing them in the direct mapped cache. When a large block is replaced in the
spatial buffer, small blocks belonging to it that have been accessed at least once
during its residency are moved into the direct mapped cache. This approach
effectively reduces conflict misses and thrashing at the same time.

Making the block size of direct mapped cache as small as possible

improves temporal locality by increasing the number of available entries for a
given cache space. For example, the number of entries for four byte block is 16
times more than for 64 byte blocks. Therefore, the lifetime of a data item in a
direct mapped cache with s – byte block size is at most l/s times that for a
cache with l byte (l>s) block size, for a given cache size. In conventional direct
mapped cache the increase in spatial locality due to a larger block size gives a
greater performance improvement than the increase in temporal locality
resulting from increasing the number of entries. However, the spatial buffer of
the SMI cache provides the necessary spatial locality and thus we choose the
smallest block size possible for the direct mapped cache.

Fetching a large block when a cache miss occurs increase spatial

locality. Our simulations show the probability that neighboring small blocks
will be accessed during the life time of initial blocks is more than 50% for most
benchmarks. An intelligent prefetching approach that emphasizes prefetching
of data that has exhibited evidence of spatial locality can also improve cache
performance.

Instead of loading a missed block directly into the direct mapped

cache, we load it into the fully associative spatial buffer. Then, time interval
proportional to the number of entries in the buffer elapsed before it is evicted.
The small blocks, which are individual words in the simple case within the
large block, that have been referenced during that time are moved into the
direct mapped cache when the large block is evicted.

2.2: SMI Cache System structure:

The direct mapped cache is the main cache and its organization
is similar to a traditional direct mapped cache. But with a smaller block size.
The spatial buffer is designed so that each entry is a collection of several
banks, each of which is the size of a block in the direct mapped cache. The tag
space of the buffer is a content addressable memory (CAM). The small blocks
in each bank include a hit bit (H), to distinguish referenced blocks from
unreferenced blocks. Each large block entry in the spatial buffer further has a
prefetch bit (P), which direct the operation of prefetch controller. The prefetch
bits are used to dynamically generate prefetch operations.

When CPU performs a memory reference, the direct mapped cache

and the spatial buffer are searched in parallel. A hit in the direct mapped cache
is processed is the same way as a hit in the conventional L1 cache. When a
miss occurs in both the direct mapped cache and the spatial buffer, a large
block is fetched into the spatial buffer. If a reference misses in the direct
mapped cache, but hits in the spatial buffer, its corresponding small block is
simply fetched from the spatial buffer and the hit bit for that small block is set
at the same time.

The prefetch controller generates a prefetch signal when a large

block is accessed and found to have multiple hit bits set. Two operations are
performed consecutively by the prefetch controller. Given a hit in the ith large
block, the first operation search the tags of the spatial buffer to direct whether
the (i+1)th large block is already present. A one cycle penalty occurs in this
case, but the overhead is negligible because prefetching is initiated in response
to only about 1.5 ~2.5 % of the total number of addresses generated by the
CPU. Thus the average MCPI is increased by about 0.06 %.If the (i+1)th large
block is not already in the spatial buffer, the second operation is performed: It
is faced into a prefetched buffer and the P bit in the ith large block is set to
prevent it from generating further prefetches. The number of prefetch buffer
entries is assumed to be one.

If misses occurs in both the direct mapped cache and the

spatial buffer, the cache control initiates miss handling. When this occurs, a
block that was already in prefetch buffer is transferred to spatial buffer.
Therefore, the transfer time is hidden because 19 clock cycles are required for
handling a miss. But, the missed block may already be present in the prefetch
buffer. Therefore when the block in the prefetch buffer is transferred to the
spatial buffer the tag value should be simultaneously compared with the
generated address. This comparison can be implemented either by using the
comparator in the direct mapped cache or an additional comparator. If the
comparator shows a match, the requested data item is transmitted to the CPU
and transferred to the spatial buffer at the same time. Also the ongoing miss
signal should be cancelled by the cache controller

2.3: Operational Model of The SMI Cache:

The algorithm for managing the SMI cache is as follows.

Prefetching is performed dynamically depending upon both H and P bits for a
particular large block in the spatial buffer. On every memory access, both the
direct mapped cache and the spatial buffer are accessed at the same time. The
different cases for the operational model are as follows.

1. Case of Cache Hits:

On every memory access, a hit may occurs at the direct mapped

cache or the spatial buffer. First, consider the case of a hit in the direct mapped
cache. If a read access to the direct mapped cache is a hit, the requested data
item is transmitted to the CPU without delay. If a write access to the direct
mapped cache is a hit, the write operation is performed and the dirty bit for the
block is set.

In second case when the hit is in the spatial buffer, the

data in the small block are sent to the CPU and the hit bit is set to the mark it
as referenced. As an aside, we also reduce power consumption by using the
most significant bits of the large block offset to activate just one of the banks
in the spatial buffer. For example, if the size of a small block is 8-byte and a
large block is 32-byte, there are four banks in the spatial buffer. When a large
block is accessed and its P bit is in the reset state, a prefetch operation is
initiated if a hit occurs in any bank of the spatial buffer and one or more hit
bits are already set. At the same time, the tags of the spatial buffer are
searched for the prefetch address to check whether it is already present, in
which case the prefetch is squashed and the P bit is set. If the address is not in
the spatial buffer, the prefetch controller generates the prefetch signal and the
target address to fetch the large block into the prefetch buffer from the next
level of memory. At the same time, the P bit of the original large block is set to
prevent the repetition of the prefetch. If the P bit of the large block is set, the
consecutive large block must be present in either the spatial buffer or the
prefetch buffer, and there is no need to search the tags within the spatial
buffer.

If a large block is in the prefetch buffer when a

subsequent prefetch signal is generated as a result of a cache miss, then the
miss stalls while the content of the prefetch buffer are moved into the spatial
buffer. According to simulation result, this case almost never occurs, even for
a prefetch buffer with one entry. This is because the overall rate of prefetch
operation is only about 0.3% and the miss-ratio is about 1.7 percent.
Therefore, the probability for a prefetch to be initiated in this manner is about
six times smaller than the miss ratio. Because the utilization of the prefetch
block is over 90 percent, it is not worth adding hardware specifically to handle
this rare case, relying instead on the existing cache stall mechanism.

2. Case of Cache Misses:

If a miss occurs in both the caches, a large block including

the missed small block is brought in the spatial buffer from the next level of
memory. Let an 8 K.B. direct mapped cache with a small block size of 8 bytes
and a 1 K.B. spatial buffer with a large block size of 32 bytes, so four
sequential blocks are contained within a 32 byte block. There are further two
cases.

(a) The Spatial Buffer is not full:

If at least one entry in the spatial buffer is in the invalid state, a large
block is fetched and stored in the spatial buffer. When a particular small block
is accessed by the CPU, the corresponding hit bit is set to one. Thus, the hit bit
of the small block identifies it as a referenced block.

(b) The Spatial Buffer is full:

If the spatial buffer is full, the oldest entry is replaced according

to a FIFO policy. At that point, the blocks in the entry whose hit bits are set are
moved into the direct mapped cache. Because these actions are accomplished
while the cache controller is handling a miss, this operation does not introduce
any additional delay.

Cache write back does not occur from the spatial buffer because
any modified or referenced small block is always moved to the direct mapped
cache or victim cache, which would typically have the small block size (e.g.,
32 bytes) as the spatial buffer, write back must be performed for the full 32
byte block, even though one word require write back. In contrast, the SMI
cache executes the write back operation only for the marked 8-byte small
blocks. Therefore, write traffic into memory is potentially reduced to a
significant degree.
 The potential exists in any split cache for incoherent copies of
blocks appear in the different subcaches. To avoid this problem, we chose and
simulated a simple mechanism, which is as follows: When a global miss
occurs, the cache controller searches the tags of the temporal caches to detect
whether any of the four small blocks belonging to the particular large block
being fetched are present in the temporal caches. If a match is detected, then all
the corresponding small blocks in the temporal cache are invalidated. Each of
these small blocks that is also dirty is than used to update its corresponding
entry in the spatial buffer once the large block has been loaded. This search
operation can be accomplished while the cache controller is handling a miss.
Further, the power consumption is negligible because the miss ratio is only
about 1.7% of the total number of the addresses generated by the CPU.

A small block may thus temporarily exist in the temporal

cache in the invalid state, while its valid copy is being referenced in the spatial
buffer. When its corresponding large block is replaced, the small block is
copied into the temporal cache once again. Therefore, there is almost no
performance decrease. Of course, if three or four small blocks are present in
both the temporal cache and the spatial cache, then the effective utilization of
total cache space decreases a bit more, but is still negligible. This mechanism
also applies in the case of transferring a prefetched block into the spatial buffer.

3: Performance Evaluation
 The direct mapped cache is chosen, for comparison in
terms of performance and cost.

3.1: Time of Prefetch Signal Generation and Overhead:

The more meaningful measure to evaluate the performance of

any given memory hierarchy is the average memory access time:

Average memory access time=

Hit time + Miss rate * Miss penalty

Here, hit time is the time to process a hit in the cache and miss penalty is the
additional time to service the miss. The basic parameters for the simulation are
as follows: The hit time of the direct mapped cache and fully associative buffer
are both assumed to be one cycle. We assumed 15 cycles are needed for a miss.
Therefore, each 8-byte block is transferred from the off-chip memory after a 15
cycle penalty. These parameters are based on the values for common 32-bit
embedded processor (e.g., Hitachi SH4 or ARM920T).

Simulations show s that prefetching when the number of hit bits

is two achieves a more significant miss gain than the other case, in spite of the
potential for greater overhead due to increased prefetching frequency. With
respect to power consumption, memory traffic, and the accuracy of the
prefetching operation, the “Prefetch-4” mechanism provides the most
significant effect.
Finally in case of prefatching a block that does not exist in the special
buffer, the target block is simply fetched into the prefetch buffer. The rate at
which prefetch actually occurred and that prefetched blocks are actually
referenced is shown in fig 4 and fig 5 respectively. With only a small number
of prefetch operations, the SMI cache system achieves a significant
performance gain with low overhead. For the “prefetching-4” mechanism, the
utilization of the prefetched blocks is over 90%. This data clearly show that
spatial locality is enhanced by prefatching a neighboring block intelligently
when a spatial buffer hit occurs.

3.2: Comparison of a Conventional Cache with the SMI

Cache Configuration

Two common performance metrics, the miss ratio and the average
memory access time, are used to evaluate and compare an SMI cache system
operating in a “prefetch-4” configuration with other approaches. Here, the
direct mapped cache is compared with the SMI cache in terms of miss ratio and
average memory access time.

Miss Ratio and Average Memory Access Time

The combination of an 8-byte small block
and a 32-byte large block shows the best performance for most cases.

The cache miss ratio for the conventional direct-mapped cache

and the SMI cache is shown in the Table-A. For the direct mapped cache,
denoted as DM, the notation “32KB-32byte” denotes a 32KB direct mapped
cache with a block size of 32bytes. The SMI cache notation “8K8-1K32”
denotes an 8KB direct-mapped cache with a block size of 8bytes and a 1KB
spatial buffer with a block size of 32-bytes. The average miss ratio of the SMI
cache for a given size (e.g., 8KB) is equal to a conventional direct mapped
cache with a cache size of four or eight time as much space (e.g., 32 or 64KB)
in nonprefetching mode and prefetching mode, respectively. Moreover,
increasing the spatial buffer space achieves a more significant performance
gain than increasing the direct mapped cache space. The miss ratios for a
conventional two way set associative cache and the SMI cache are compared in
Table-B. The two-way set associative cache greatly reduces the miss ratio, but
because of its slower access time and higher power consumption, this is not
much effective. The result shows that the SMI cache can achieve better
performance than a two way set associative cache with double the space.
 The average memory access time for the conventional
direct mapped cache and the SMI cache are compared in Table-C. We can see
that 8K8-1K32(Prefetch mode) cache has greater Average Memory Access
Time (cycle) than the Direct mapped cache, while 8K8-2K32(Prefetch mode)
has less Average Memory Access Time (Cycle). This analysis shows that
application with a high degree of locality, show an especially strong
performance improvement with the SMI cache.

4.3: Relation between Cost and Performance

In general the, logic to manage the tags for the fully associative
cache is designed as a CAM structure for simultaneous comparison for each
entry. Because each CAM cell is a combination of storage and comparison, the
size of CAM cell is double of RAM cell. For fair performance/cost analysis,
the performance for various direct mapped-cache and buffer size is evaluated.
The metric is rbe (register bit equivalent) and the total area can be calculated
as follows:

Area = PLA + RAM + CAM. (1)

Here, the control logic PLA (Programmable logic array) is assumed to be 130
rbe, a RAM cell as 0.6 rbe, and a CAM cell is 1.2 rbe. Equation (2) represents
the RAM area:

RAM = 0.6 * (#entries + Lsense_amp) *

((#data_bits + #status_bits) + Wdriver), (2)

where Lsense_amp is the bit length of a bit line sense amplifier, Wdriver
the data width of a driver, #entries the number of rows of the tag array or data
array, #data_bits the tag bit or data bit of one set, and #status_bits the state bits
of one set. Finally, (3) calculate the area of the CAM:

CAM = 0.6 * (2^ (1/2) * #entries + Lsense_amp) *

(2^ (1/2) * #tag_bits + Wdriver), (3)

where #tag_bits is the number of bits for one set in the tag array. Table 1
shows the performance/cost ratio for direct mapped cache and SMI cache.

The SMI cache shows about a 60% area reduction compared with the 64KB-
32byte conventional direct mapped cache, even though it provides higher
performance. And, it offers an 80 percent area reduction compared with the
64KB-32byte configuration, while providing much higher performance. Also
the improvement ratio for the average memory access time shows that the
8KB-2KB SMI is the best configuration.

Performance and Cost of the SMI cache and Direct-mapped Cache

32KB-32byte 8KB-1KB 64KB-2KB 8KB-2KB

(DM) (SMI) (DM) (SMI)
Area 177496 rbe 67431 rbe 352596 rbe 73680 rbe
(Improvement ratio) (1.00) (0.38) (1.99) (0.42)

Miss ratio 1.89% 1.61% 1.46% 1.37%

(Improvement ratio) (1.00) (0.85) (0.77) (0.73)

Avg. memory 1.34 cycle 1.29 cycle 1.26 cycle 1.25 cycle
access time (1.00) (0.96) (0.94) (0.93)
(Improvement ratio)

Miss Ratio (%)

 32KB-32byte (DM) 1.90
64KB-32 byte (DM) 1.45
8K8-1K32 (NPF) 2.05
8K8-2K32 (NPF) 1.80
8K8-1K32 (PF) 1.70
8K8-2K32 (PF) 1.45

Table-A: Miss Ratio of the Direct Cache and SMI Cache.

Miss Ratio (%)

16KB-32byte (2-way) 2.10

32KB-32 byte (2-way) 1.10
8K8-1K32 (NPF) 2.05
8K8-2K32 (NPF) 1.80
8K8-1K32 (PF) 1.70
8K8-2K32 (PF) 1.45

Table-B: Miss Ratio if the Two-way set associative cache and SMI cache.

Average Memory Access Time (Cycle)

32KB-32byte (DM) 1.35

64KB-32 byte (DM) 1.28
8K8-1K32 (NPF) 1.36
8K8-2K32 (NPF) 1.31
8K8-1K32 (PF) 1.28
8K8-2K32 (PF) 1.25

Table-C: Average memory access time of the direct mapped cache and the SMI
cache.
 4. Conclusion

To design a simple but high performance cache system with low

cost , a new caching mechanism for exploiting two types of locality
effectively and adaptively is designed: A direct mapped cache with a
small block size for exploiting temporal locality and a fully associative
spatial buffer with a large block size for exploiting spatial buffer. An
intelligence hardware based prefetching mechanism is used to maximize
the effect of spatial locality.

The SMI cache overcomes the structural drawbacks of

direct mapped caches, such as conflict misses and thrashing. We can
evaluate the SMI cache system in two configurations, nonprefetching
mode and prefetching mode, to analyze the contribution of intelligent
prefetching. Both modes provide high performance, but non prefetching
mode offers lower power consumption, while prefetching mode offers
higher performance.

The temporal mechanism of the SMI cache

decreases conflict misses by about 26% and the spatial locality miss
ratio decrease by about 48%.

Table of Content:
1. Introduction
 2. Selective Mode Intelligent Cache System

2.1. Motivation

2.2. SMI Cache System Structure

2.3. Operational Model of The SMI cache

2.3.1. Case of Cache Hits

2.3.2. Case of Cache misses

(A). The spatial buffer is not full:

(B). The spatial buffer is full:

3. Performance Evaluation

3.1. Time of Prefetch Signal Generation and Overhead

3.2. Comparison of a Conventional Cache with the SMI
Cache Configuration (Miss Ratio and Average
Memory Access Time)
3.3. Relation between Cost and Performance

4. Conclusion

References:
1. IEEE transaction On computers,
 Vol. 52, NO.5, MAY 2003
2. Computer Architecture and Organization:
John P. Hayes
3. Computer Organization and Architecture:
William Stallings