> For the complete documentation index, see [llms.txt](https://blog.ruipan.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://blog.ruipan.xyz/earlier-readings-and-notes/cs759-hpc-course-notes/lecture-4-the-memory-hierarchy.-caches..md).

# Lecture 4: The memory hierarchy. Caches.

## Lecture Summary

* Execution times
* Memory related issues
* The memory hierarchy
* Caches

## Execution Times - Nomenclature

* Wall Clock Time: Amount of time from the beginning to the end of a program
* CPU Execution Time: Amount of time on the CPU that's dedicated to your program, requires a profiling tool to access
  * User Time: Time spent processing instructions compiled out of code generated by the user or in libraries that are directly called by user code
  * System Time: Time spent in support of the user’s program but in instructions that were not generated out of code written by the user (e.g., OS support for opening/reading a file, throwing an exception, etc.)
* Clock cycle: The length of the period for the processor clock (e.g., a 1GHz processor has a clock cycle of 1 nanosecond)
* The CPU Performance Equation: CPU Execution Time = Instruction Count \* Clock-Cycles per Instructions (CPI) \* Clock Cycle Time = Instruction Count \* Clock-Cycles per Instructions (CPI) / Clock Rate

![The SPEC CPU benchmark. CPI<1: Multiple-issue is in play. For combinational optimization, there are probably a lot of pipeline stalls](https://1313833672-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MMTslgmrrtRXvxD2lk9%2F-MSTXpt8jD69qcMZPGPz%2F-MSTe-MQNFNzSDIsyTCN%2FScreen%20Shot%202021-02-01%20at%2011.33.19%20AM.png?alt=media\&token=692348c5-3918-49aa-8a34-d46b915882f7)

## Memory & Cache

* SRAM (Static Random Access Memory): Expensive but fast (short access time), bulky, transistor hog, needs no refresh
* DRAM (Dynamic \~): Cheap but slow, information stored as a charge in a capacitor, higher capacity per unit area, needs refresh every 10-100ms, sensitive to disturbances

![](https://1313833672-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MMTslgmrrtRXvxD2lk9%2F-MSTXpt8jD69qcMZPGPz%2F-MSU96h6oBJjJDyh3113%2FScreen%20Shot%202021-02-01%20at%201.53.41%20PM.png?alt=media\&token=b76f74e4-d9ce-4694-a512-209ac14b8133)

![](https://1313833672-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MMTslgmrrtRXvxD2lk9%2F-MSTXpt8jD69qcMZPGPz%2F-MSU9jbjU9RjdSz3xbQa%2FScreen%20Shot%202021-02-01%20at%201.56.23%20PM.png?alt=media\&token=43201788-7b83-413c-b2c6-822b5aa63167)

The memory hierarchy (the pyramid of tradeoffs):

* A dedicated hardware asset called MMU (Memory Management Unit) is used to manage the hierarchy
* Tradeoff:
  * DRAM off-chip: Main memory
  * SRAM on-chip: Cache
    * Caches have a deeper hierarchy: L1+L2+L3. L1 is faster and smaller than L2 & L3.
    * Different types of caches
      * Data caches: Feeds processor with data manipulated during execution
      * Instruction caches: Stores instructions
    * The ratio between cache size & main memory size: \~1:1000

![](https://1313833672-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MMTslgmrrtRXvxD2lk9%2F-MSTXpt8jD69qcMZPGPz%2F-MSUABEmaiBF07xEoJNU%2FScreen%20Shot%202021-02-01%20at%201.58.22%20PM.png?alt=media\&token=aecb6fa3-a4cc-4fa9-a0d1-a0250f1f88cc)

![](https://1313833672-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MMTslgmrrtRXvxD2lk9%2F-MSTXpt8jD69qcMZPGPz%2F-MSUAFE_Akzl9ZaP5-5u%2FScreen%20Shot%202021-02-01%20at%201.58.38%20PM.png?alt=media\&token=0bfd735a-caf5-4cc4-a89e-b91b5569f024)

The reason why cache works is the principle of locality: Programs tend to use data and instructions with addresses near or equal to those they have used recently.

* Temporal locality: Recently referenced items are likely to be referenced again in the near future
  * Data references: For example, in the code snippet below, the variable sum gets referenced at each iteration
  * Instruction references: The loop is cycled through repeatedly
* Spatial locality: Items with nearby addresses tend to come into use together
  * Data references: The elements in the array abc are accessed in succession (stride-1 reference pattern)
  * Instruction references: The instructions are referenced in sequence

```
sum = 0;
for (i = 0; i < n; i++)
    sum += abc[i];
return sum;
```

### Case study: Adding the entries in an N-dimensional matrix (not covered in class)

Take-home message: Well-written programs leverage data/instruction locality (which brings cache into the play) for better performance