Transformer Models
Transformer models are neural network architectures that use self-attention to process sequences in parallel rather than step by step. They power modern language models, machine translation, and many other tasks where understanding relationships between elements in a sequence is critical.
itArtificial intelligence and machine learning | OpenSkills.info
Intro
Transformer Models
The transformer is the neural network architecture behind every major large language model — GPT, Claude, Gemini, Llama, and the rest. Introduced in the 2017 paper "Attention Is All You Need," it replaced the recurrent and convolutional architectures that came before it, and it did so by discarding an assumption those architectures had never questioned: that a model needs to read text one token at a time, in order.
This course goes deeper into that architecture than a general orientation to LLMs does. If you haven't already, the LLM Foundations course in this catalog covers what a large language model is, how it's trained, and what it can and can't do — the mental model this course builds on. Here, the focus narrows to the mechanism itself: self-attention, multi-head attention, positional encoding, and the block structure that everything else is built from.
Why transformers replaced RNNs and CNNs
Before the transformer, sequence models were recurrent: a recurrent neural network (RNN) processed a sentence one token at a time, carrying forward a hidden state that summarized everything seen so far. That design had two structural problems. It couldn't be parallelized — token 50 couldn't be processed until tokens 1 through 49 were done — which made training slow. And it struggled with long-range dependencies, since information from early in a long sequence had to survive being compressed through dozens of sequential updates before it could influence a later token.
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://arxiv.org/abs/1706.03762
Supports
- The transformer is based solely on attention mechanisms, dispensing with recurrence and convolutions entirely
- Self-attention computes relationships between all positions in a sequence in parallel
- Multi-head attention runs multiple attention computations in parallel to capture different relationship types
- The architecture is more parallelizable and requires significantly less time to train than recurrent architectures
- Each encoder/decoder layer contains a feed-forward network in addition to attention sub-layers
- Residual connections around each sub-layer followed by layer normalization
- The original design is an encoder-decoder architecture aimed at sequence-to-sequence tasks like translation
- https://huggingface.co/learn/llm-course/chapter1/4
Supports
- Transformers use an encoder-decoder structure where the encoder builds a representation of input and the decoder generates a target sequence
- Encoder-only models (BERT) excel at understanding tasks such as classification and entity recognition
- Decoder-only models (GPT) focus on generation, predicting one token at a time
- Encoder-decoder models (T5) perform well on generative tasks requiring input context, such as translation or summarization
- https://huggingface.co/learn/llm-course/chapter1/5
Supports
- Most transformer implementations use variants of encoder-only, decoder-only, or encoder-decoder configurations
- Vision Transformer (ViT) converts images into patch sequences processed like text tokens
- Whisper converts raw audio to spectrograms, then processes them through an encoder-decoder architecture
- Encoder-decoder models like BART handle sequence-to-sequence work such as summarization
- https://huggingface.co/blog/designing-positional-encoding
Supports
- Self-attention is a set operation and is permutation equivariant, treating tokens identically regardless of position
- Positional encodings are added to token embeddings to indicate position, since transformers process all tokens in parallel
- Desirable properties for a positional encoding scheme include consistency across sequence lengths, generalization to longer sequences, and clean combination with attention
- Rotary Positional Encoding (RoPE) is the current widely adopted approach, encoding relative position by rotating query and key vectors as a function of position