Transformer Models

The transformer is the neural network architecture behind every major large language model — GPT, Claude, Gemini, Llama, and the rest. Introduced in the 2017 paper "Attention Is All You Need," it replaced the recurrent and convolutional architectures that came before it, and it did so by discarding an assumption those architectures had never questioned: that a model needs to read text one token at a time, in order.

This course goes deeper into that architecture than a general orientation to LLMs does. If you haven't already, the LLM Foundations course in this catalog covers what a large language model is, how it's trained, and what it can and can't do — the mental model this course builds on. Here, the focus narrows to the mechanism itself: self-attention, multi-head attention, positional encoding, and the block structure that everything else is built from.

Why transformers replaced RNNs and CNNs

Before the transformer, sequence models were recurrent: a recurrent neural network (RNN) processed a sentence one token at a time, carrying forward a hidden state that summarized everything seen so far. That design had two structural problems. It couldn't be parallelized — token 50 couldn't be processed until tokens 1 through 49 were done — which made training slow. And it struggled with long-range dependencies, since information from early in a long sequence had to survive being compressed through dozens of sequential updates before it could influence a later token.

The transformer architecture is based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Every token in the input is processed in parallel, and any token can directly relate to any other token regardless of distance — no compression through intermediate steps required. The paper's own benchmarks showed this wasn't just an engineering convenience: transformers were both more parallelizable and required significantly less time to train than the recurrent architectures they replaced, while improving translation quality at the same time.

Self-attention: the core mechanism

Self-attention is the operation that lets every token in a sequence weigh how relevant every other token is to it. For each token, the model computes a query (what this token is looking for), and every token in the sequence exposes a key (what this token offers) and a value (the information it actually carries). Comparing a token's query against every other token's key produces a relevance score for each pair; those scores are normalized and used to blend the value vectors into a new representation for that token.

Concretely, in a sentence like "the capital of France is Paris," the token generating "Paris" needs the model to have attended strongly to "France" and "capital" — self-attention is the mechanism that lets that relationship form, computed directly rather than inferred through a chain of sequential updates.

Multi-head attention runs this process several times in parallel, each with its own learned queries, keys, and values — each "head" free to specialize in a different kind of relationship: one might track subject-verb agreement, another might track coreference (which pronoun refers to which noun), another might track topical relevance across a long document. The outputs of all heads are combined, giving the model several simultaneous, independent views of how the tokens in a sequence relate to each other.

Positional encoding: putting order back in

Self-attention has a side effect worth understanding on its own: it's a set operation. Left alone, it treats a sequence's tokens the same way regardless of their order — the word "dog" attends and is attended to identically whether it's the first word in the sentence or the last, because nothing in the raw computation carries position information. That's a problem for language, where order changes meaning.

Positional encoding fixes this by injecting position information into each token's representation before it enters the attention layers. Early transformer designs used a fixed sinusoidal pattern; the field has since moved through several approaches in pursuit of encodings that stay consistent regardless of sequence length, extend to sequences longer than anything seen in training, and combine cleanly with the attention computation itself. Rotary Positional Encoding (RoPE) is the current widely adopted approach in modern LLMs, encoding relative position by rotating query and key vectors as a function of their position rather than adding a separate fixed positional vector.

The transformer block

A transformer is a stack of identical blocks, each combining three ingredients:

• Multi-head self-attention, described above — how tokens gather information from each other.
• A feed-forward network, applied identically to every token's representation after attention — where much of the model's learned knowledge is thought to live.
• Residual connections and layer normalization around each of these two sub-layers — the residual connection adds a sub-layer's input back to its output, which keeps gradients flowing through very deep stacks of blocks during training; layer normalization keeps the scale of activations stable as they pass through dozens of stacked blocks.

Stack enough of these blocks — modern frontier models run into the hundreds of layers — and the result is a model that can build increasingly abstract representations of its input at each successive layer, from surface-level token relationships near the bottom of the stack to task-relevant, long-range reasoning nearer the top.

Three architecture families

Not every transformer is built the same way. The original paper described an encoder-decoder design — an encoder that builds a full representation of the input, and a decoder that generates output conditioned on that representation — aimed at sequence-to-sequence tasks like translation. Since then, the field has split into three families, each suited to a different kind of task:

• Encoder-only models (BERT is the canonical example) build a bidirectional representation of the input, attending to context on both sides of every token. They're built for understanding tasks: classification, entity recognition, extracting structured information from text.
• Decoder-only models (GPT is the canonical example) attend only to earlier tokens and generate output one token at a time. This is the architecture behind essentially every modern chat-oriented LLM, because autoregressive generation — predicting the next token given everything so far — is exactly what conversational and text-generation use cases need.
• Encoder-decoder models (T5 and BART are examples) keep both halves, and remain a strong fit for tasks that need to fully digest an input before producing a structured, different-length output — translation and summarization in particular.

If you've used a modern LLM API, you've almost certainly been using a decoder-only model. That architecture choice — attending only to the past, generating one token forward at a time — is the reason LLM inference works the way it does, which the training and inference course in this catalog covers in more depth.

Beyond text

The same building blocks — self-attention, positional encoding, feed-forward layers, stacked blocks — turn out not to be specific to language at all. Vision Transformer (ViT) applies the identical architecture to images by splitting an image into fixed-size patches and treating each patch as if it were a token. Whisper applies an encoder-decoder transformer to audio by converting raw sound into a spectrogram representation before processing it the same way text would be processed. The architecture generalizes because self-attention is fundamentally a mechanism for relating elements of a sequence to each other — and once you can represent your input as a sequence of tokens, image patches, audio frames, whatever, the same mechanism applies.

Where this course stops

This is an orientation to the transformer architecture itself: why it replaced recurrent and convolutional models, how self-attention and multi-head attention work, why positional encoding is necessary, what a transformer block is built from, and the three architecture families that came out of it. It does not derive the underlying mathematics in full or walk through implementing a transformer from scratch. For the training pipeline and inference mechanics that run on top of this architecture, see the large language model training and inference course in this catalog.