Memory Hierarchy

Memory Hierarchy is an enhancement to organize the memory such that

it can minimize the access time. The Memory Hierarchy was developed
based on a program behavior known as locality of references.The figure
below clearly demonstrates the different levels of memory hierarchy :

This Memory Hierarchy Design is divided into 2 main types:

1. External Memory or Secondary Memory –
Comprising of Magnetic Disk, Optical Disk, Magnetic Tape i.e.
peripheral storage devices which are accessible by the processor via
I/O Module.
2. Internal Memory or Primary Memory –
Comprising of Main Memory, Cache Memory & CPU registers.
This is directly accessible by the processor.
We can infer the following characteristics of Memory Hierarchy Design
from above figure:
1. Capacity:
It is the global volume of information the memory can store. As we
move from top to bottom in the Hierarchy, the capacity increases.
 2. AccessTime:
It is the time interval between the read/write request and the
availability of the data. As we move from top to bottom in the
Hierarchy, the access time increases.
3. Performance:
Earlier when the computer system was designed without Memory
Hierarchy design, the speed gap increases between the CPU
registers and Main Memory due to large difference in access time.
This results in lower performance of the system and thus,
enhancement was required. This enhancement was made in the form
of Memory Hierarchy Design because of which the performance of
the system increases. One of the most significant ways to increase
system performance is minimizing how far down the memory
hierarchy one has to go to manipulate data.
4. Cost per bit:
As we move from bottom to top in the Hierarchy, the cost per bit
increases i.e. Internal Memory is costlier than External Memory.

Observations-
 
The following observations can be made when going down in the
memory hierarchy-
The goals of memory hierarchy are-
5. To obtain the highest possible average access speed
6. To minimize the total cost of the entire memory system

Inclusion, Coherence, and Locality

Inclusion Property

if an information word is found in Mi, then copies of the same word can
be also found in all upper levels Mi+1, Mi+2, …, Mn. However, a word
stored in Mi+1 may not be found in Mi. A word miss in Mi implies that
it is also missing from all lower levels Mi-1, Mi-2, …, M1. The highest
level is the backup storage, where everything can be found.
Information transfer between the CPU and cache is in terms of words (4
or 8 bytes each depending on the word length of a machine. The cache
(M1) is divided into cache blocks, also called cache lines by some
authors. Each block is typically 32 bytes
(8 words). Blocks are the units of data transfer between the cache and
main memory.

The main memory (M2) is divided into pages, say, 4Kbytes each. Each
page contains 128 blocks. Pages are the units of information transferred
between disk and main memory.

Coherence Property
The coherence property requires that copies of the same information
item at successive memory levels be consistent. If a word is modified in
the cache, copies of that must be updated immediately or eventually at
all higher levels. The hierarchy
should be maintained as such. Frequently used information is often
found in the lower levels in order to minimize the effective access time
of the memory hierarchy. In general, there are two strategies for
maintaining the coherence in a memory hierarchy.

Write Through:

The first method is called write-through (WT), which demands

immediate update in Mi+1 of a word is modified in Mi, for i = 1, 2, ….,
n-1.
Write Back
The second method is write-back (WB), which delays the update in
Mi+1 until the word being modified in Mi is replaced or removed from
Mi.
Locality of References

The memory hierarchy was developed based on a program behavior

known as locality of references. Memory references are generated by the
CPU for either instruction or data access. These accesses tend to be
clustered in certain regions in time, space, and ordering.
In other words, most programs act in favor of a certain portion of their
address space at any time window. Hennessy and Patterson (1990) have
pointed out a 90-10 rule which states that a typical program may spend
90% of its execution time on only 10% of the code such as the innermost
loop of a nested looping operation.
There are three dimensions of the locality property: temporal, spatial,
and sequential. During the lifetime of a software process, a number of
pages are used dynamically. The references to these pages vary from
time to time; however, they follow certain access patterns. These
memory reference patterns are caused by the following locality
properties:
1. Temporal locality-Recently referenced items (instruction or data) are
likely to be referenced again in the near future. This is often caused by
special program constructs such as iterative loops, process stacks,
temporary variables, or subroutines. Once a loop is entered or a
subroutine is called, a small code segment will be referenced repeatedly
many times. Thus temporal tends to
cluster the access in the recently used areas.
2. Spatial locality-This refers to the tendency for a process to access
items whose addresses are near one another. For example, operations on
tables or arrays involve accesses of a certain clustered area in the
address space. Program segments, such as routines and macros, tend to
be stored in the same neighborhood of the memory space.
3. Sequential locality-In typical programs, the execution of instructions
follows a sequential order (or the program order) unless branch
instructions create outof- order executions. The ratio of in-order
execution to out-of-order execution is roughly 5 to 1 in ordinary
programs. Besides, the access of a large data array also follows a
sequential order.

Hit: data is found at higher level – the ideal case – Operation performed
quickly §
Miss: data not found –
Must continue the search at the next level down
– After data is eventually located, it is copied at the memory level where
the miss happened
 Hit rate (hit ratio): fraction of accesses that are hits at a given level of
the hierarchy
Hit time: Time required to access a level of the hierarchy, including
time to determine whether access is a hit or miss
Miss rate (miss ratio): fraction of accesses that are misses at a given
level (= 1 – hit rate)
§ Miss penalty: Extra time required to fetch a block into some level
from the next level down
Hit ratio = hit / (hit + miss) = no. of hits/total accesses

Miss ratio= 1-hit ratio

 Memory is organized at different levels.

 CPU may try to access different levels of memory in different ways.
 On this basis, the memory organization is broadly divided into two types-
 

1. Simultaneous Access Memory Organization

2. Hierarchical Access Memory Organization
 

1. Simultaneous Access Memory Organization-

 

In this memory organization,

 All the levels of memory are directly connected to the CPU.

 Whenever CPU requires any word, it starts searching for it in all the levels simultaneously.
 
 Example-01:
 

Consider the following simultaneous access memory organization-

Here, two levels of memory are directly connected to the CPU.

Let-

 T1 = Access time of level L1

 S1 = Size of level L1
 C1 = Cost per byte of level L1
 H1 = Hit rate of level L1
Similar are the notations for level L2.

Average Memory Access Time-

 

Average time required to access memory per operation

= H1 x T1 + (1 – H1) x H2 x T2

= H1 x T1 + (1 – H1) x 1 x T2

= H1 x T1 + (1 – H1) x T2

Important Note
In any memory organization,
 The data item being searched will definitely be present in the last level.
 Thus, hit rate for the last level is always 1.

 
 Average Cost Per Byte-
 

Average cost per byte of the memory

= { C1 x S1 + C2 x S2 } / { S1 + S2 }

Example-02:
 

Consider the following simultaneous access memory organization-

Here, three levels of memory are directly connected to the CPU.

Let-

 T1 = Access time of level L1

 S1 = Size of level L1
 C1 = Cost per byte of level L1
 H1 = Hit rate of level L1
Similar are the notations for other two levels.

Average Memory Access Time-

 

Average time required to access memory per operation

= H1 x T1 + (1 – H1) x H2 x T2 + (1 – H1) x (1 – H2) x H3 x T3

 = H1 x T1 + (1 – H1) x H2 x T2 + (1 – H1) x (1 – H2) x 1 x T3

= H1 x T1 + (1 – H1) x H2 x T2 + (1 – H1) x (1 – H2) x T3

Average Cost Per Byte-

 

Average cost per byte of the memory

= { C1 x S1 + C2 x S2 + C3 x S3 } / { S1 + S2 + S3 }

2. Hierarchical Access Memory Organization-

 

In this memory organization, memory levels are organized as-

 Level-1 is directly connected to the CPU.

 Level-2 is directly connected to level-1.
 Level-3 is directly connected to level-2 and so on.
 

Whenever CPU requires any word,

 It first searches for the word in level-1.

 If the required word is not found in level-1, it searches for the word in level-2.
 If the required word is not found in level-2, it searches for the word in level-3 and so on.
 

Example-01:
 

Consider the following hierarchical access memory organization-

 
 Here, two levels of memory are connected to the CPU in a hierarchical fashion.

Let-

 T1 = Access time of level L1

 S1 = Size of level L1
 C1 = Cost per byte of level L1
 H1 = Hit rate of level L1
Similar are the notations for level L2.

Average Memory Access Time-

 

Average time required to access memory per operation

= H1 x T1 + (1 – H1) x H2 x (T1 + T2)

= H1 x T1 + (1 – H1) x 1 x (T1 + T2)

= H1 x T1 + (1 – H1) x (T1 + T2)

Average Cost Per Byte-

 

Average cost per byte of the memory

= { C1 x S1 + C2 x S2 } / { S1 + S2 }

Example-02:
 

Consider the following hierarchical access memory organization-

 
  

Here, three levels of memory are connected to the CPU in a hierarchical fashion.

Let-

 T1 = Access time of level L1

 S1 = Size of level L1
 C1 = Cost per byte of level L1
 H1 = Hit rate of level L1
Similar are the notations for other two levels.

Average Memory Access Time-

 

Average time required to access memory per operation

= H1 x T1 + (1 – H1) x H2 x (T1 + T2) + (1 – H1) x (1 – H2) x H3 x (T1 + T2 + T3)

= H1 x T1 + (1 – H1) x H2 x (T1 + T2) + (1 – H1) x (1 – H2) x 1 x (T1 + T2 + T3)

= H1 x T1 + (1 – H1) x H2 x (T1 + T2) + (1 – H1) x (1 – H2) x (T1 + T2 + T3)

Average Cost Per Byte-

 

Average cost per byte of the memory

 = { C1 x S1 + C2 x S2 + C3 x S3 } / { S1 + S2 + S3 }

PRACTICE PROBLEMS BASED ON MEMORY ORGANIZATION-

 

Problem-01:
 

What is the average memory access time for a machine with a cache hit rate of 80% and cache access
time of 5 ns and main memory access time of 100 ns when-

1. Simultaneous access memory organization is used.

2. Hierarchical access memory organization is used.
 

Solution-
 

Part-01: Simultaneous Access Memory Organization-

 

The memory organization will be as shown-

Average memory access time

= H1 x T1 + (1 – H1) x H2 x T2

= 0.8 x 5 ns + (1 – 0.8) x 1 x 100 ns

= 4 ns + 0.2 x 100 ns

= 4 ns + 20 ns

= 24 ns

Part-02: Hierarchical Access Memory Organization-

 
 The memory organization will be as shown-

Average memory access time

= H1 x T1 + (1 – H1) x H2 x (T1 + T2)

= 0.8 x 5 ns + (1 – 0.8) x 1 x (5 ns + 100 ns)

= 4 ns + 0.2 x 105 ns

= 4 ns + 21 ns

= 25 ns

Problem-02:
 

A computer has a cache, main memory and a disk used for virtual memory. An access to the cache takes
10 ns. An access to main memory takes 100 ns. An access to the disk takes 10,000 ns. Suppose the
cache hit ratio is 0.9 and the main memory hit ratio is 0.8. The effective access time required to access a
referenced word on the system is _______ when-

1. Simultaneous access memory organization is used.

2. Hierarchical access memory organization is used.
 

Solution-
 

Part-01:Simultaneous Access Memory Organization-

 

The memory organization will be as shown-

 

Effective memory access time

= H1 x T1 + (1 – H1) x H2 x T2 + (1 – H1) x (1 – H2) x H3 x T3

= 0.9 x 10 ns + (1 – 0.9) x 0.8 x 100 ns + (1 – 0.9) x (1 – 0.8) x 1 x 10000 ns

= 9 ns + 8 ns + 200 ns

= 217 ns

Part-02: Hierarchical Access Memory Organization-

 

The memory organization will be as shown-

 
 

Effective memory access time

= H1 x T1 + (1 – H1) x H2 x (T1 + T2) + (1 – H1) x (1 – H2) x H3 x (T1 + T2 + T3)

= 0.9 x 10 ns + (1 – 0.9) x 0.8 x (10 ns + 100 ns) + (1 – 0.9) x (1 – 0.8) x 1 x (10 ns + 100 ns + 10000 ns)

= 9 ns + 8.8 ns + 202.2 ns

= 220 ns