GPU Architecture

A graphics processing unit, or GPU, is a processor built to run many similar operations in parallel. Its architecture combines groups of execution lanes, fast on-chip memory, caches, and specialized graphics or matrix hardware to favor throughput over fast completion of one instruction stream.

itComputer architecture and hardware

Don't Panic — GPU Architecture

A GPU is a processor built to keep a great many similar operations moving at once. It is not a CPU with more enthusiasm. A CPU spends considerable hardware on finishing a few instruction streams quickly. A GPU spends more of it on parallel execution, which is excellent news for pixels, matrix elements, and other large, regular collections. It is less exciting for tiny jobs, serial dependencies, and data that must keep commuting between host and device.

The first useful map is the execution hierarchy. The host submits a kernel. The launch becomes blocks or work-groups. Hardware then issues nearby work items as warps, wavefronts, or subgroups. Those names occupy the same place in the map, but they do not promise identical widths or resources. Blocks must be independently schedulable, because the device was not designed to consult an application about which block feels emotionally ready.

Inside a compute unit, ready execution groups take turns using arithmetic, load and store, and sometimes matrix or fixed-function units. When one group waits for memory, another can issue. That is latency hiding. Occupancy describes how much resident work is available for this trick, but it is not a medal to maximize. A kernel can instead be limited by memory bandwidth, arithmetic pipelines, launch gaps, or transfers. The profiler gets the vote.

The other half of the story is the memory hierarchy. Registers are close and limited. Shared or local memory lets a work-group reuse data. Caches help without being asked. Device memory holds the larger working set, while host and peer memory live beyond an interconnect. Neighboring lanes that read nearby addresses can combine their requests. Scattered access asks for more transactions, which is a very efficient way to move less useful information more often.

Control flow has a similar wrinkle. Work items look independent in source code, but a group usually issues together. When lanes take different branch paths, the required paths run with masks and some lanes wait their turn. That is not automatically wrong. It does mean data layout and grouping affect how much useful work each issued instruction accomplishes.

For the wider explanation, open the Intro for the architecture and failure behavior. The Slides turn the hierarchy into a compact map. The Cheatsheet is the reference for vocabulary, limits, and diagnostic signals. Then use the Reference tab to move from this mental model into vendor documentation and profiling tools. The hardware will still have opinions. At least they will be measurable ones.

Don't Panic — GPU Architecture

Relevant careers