openskills.info
Course catalog
Course Preview

Memory Hierarchy

Memory hierarchy arranges registers, caches, main memory, and storage in layers with different speed, capacity, and cost. It keeps frequently used data near the processor while larger, slower layers retain the rest.

itComputer architecture and hardware

Don't Panic — Memory Hierarchy

Every processor faces the same annoying fact: no single kind of memory is fast, huge, cheap, and permanent at once. Something has to give. The memory hierarchy is the compromise computer designers settled on: stack up several kinds of storage, keep the smallest and fastest closest to the processor, and let the big slow stuff sit farther away holding everything else.

Before you picture a tidy pyramid, know that the pyramid is a teaching prop, not a blueprint. The real path runs from registers through one or more caches, into main memory, and sometimes onward to a disk or SSD standing in for memory it can't otherwise hold. Nothing forces a request to visit every layer. Most don't, and that's the entire point.

The reason this works at all is locality: programs tend to reuse the same data soon (temporal) or touch data sitting near what they just touched (spatial). A hierarchy is a bet that your program has some, and most programs do.

When a requested value is already sitting in the nearby layer, that's a hit. When it isn't, the search continues downward, and that's a miss, the same word reused, confusingly, for three different events with three different costs: a cache miss, a TLB miss (translation, not data), and a page fault (the operating system getting involved). Keeping those apart is worth the effort; conflating them is how performance debugging goes sideways.

Here's the bit that trips people up even after they've drawn the pyramid correctly: two threads can write to completely unrelated variables and still slow each other down, because the hardware doesn't move data byte by byte — it moves whole cache lines. If your two "independent" counters happen to land on the same line, the cores fight over ownership of it constantly, and the resulting slowdown looks exactly like a lock problem to anyone watching a profiler. It isn't one. It's a layout problem wearing a lock problem's clothes.

None of this is exotic wizardry reserved for kernel engineers. It's the layer beneath every "why is this slower than it should be" investigation you'll ever run, whether you're tuning a database, a game engine, or a script that felt fine until the input got large.

Where to go next: the Cheatsheet has the vocabulary and the AMAT formula in dense reference form, the Slides give you the visual map if prose isn't landing, and Field Notes is where the judgment lives, what teams get wrong about scaling versus locality, and what a stalled processor's utilization graph actually tells you. And if you want to check whether any of this stuck, the Quiz is right there waiting.

Where this skill leads

Relevant careers

See how this topic contributes to broader role-level skill maps.

Sources