Large Language Model Training and Inference

This course is the deeper companion to LLM Foundations in this catalog. That course gives you the broad orientation: what an LLM is, the training stages at a high level, and how inference works in outline. This course goes further into two specific mechanics every AI engineer eventually runs into: how the size of a training run gets decided, how a raw base model becomes an aligned assistant, and what's actually happening — and what can be tuned — every time a model generates a response.

Pretraining at scale: what the scaling laws say

Training a large language model from scratch means choosing, ahead of time, how many parameters the model will have and how many tokens of text it will train on — and those two choices interact. For years, the field's default was to make models as large as the compute budget allowed, training them on whatever data was available without treating dataset size as an equally important lever.

DeepMind's 2022 "Chinchilla" scaling law study challenged that. Training over 400 models across a range of sizes, the researchers found that for a fixed compute budget, model size and training data should be scaled together, roughly in step — and that the field's existing large models were, by that measure, undertrained: too large for the amount of data they'd seen. They demonstrated it directly: Chinchilla, a 70-billion-parameter model trained on 1.4 trillion tokens, outperformed Gopher, a 280-billion-parameter model trained on roughly 300 billion tokens, despite using the same training compute. A model a quarter of Gopher's size, trained on four times more data, won.

The practical takeaway for anyone reasoning about model capability: parameter count alone doesn't tell you what a model can do. A smaller model trained compute-optimally on enough data can outperform a much larger one that was undertrained relative to its size — which is part of why raw parameter counts became a less reliable signal of model quality over time, and why current frontier models are trained on token counts far larger than early scaling assumptions suggested was needed.

From base model to assistant: the RLHF pipeline

A freshly pretrained model is a text-completion engine, not an assistant. It predicts plausible continuations of text; it doesn't inherently know how to follow an instruction, stay on topic, or decline an unsafe request. Turning that raw capability into a usable assistant is a separate, deliberate training stage — and reinforcement learning from human feedback (RLHF) is the technique that established how to do it, first shown at scale in OpenAI's InstructGPT work.

RLHF runs in three stages, each producing the input the next stage needs:

Supervised fine-tuning (SFT). Human labelers write demonstrations of the desired behavior — example prompts paired with high-quality responses — and the base model is fine-tuned on this dataset directly, the same way any supervised fine-tuning works.
Reward model training. Labelers rank multiple model outputs for the same prompt, from best to worst. That ranking data trains a separate reward model to predict which of two outputs a human would prefer — turning subjective human judgment into a score a training loop can optimize against.
Reinforcement learning. The SFT model is further trained using reinforcement learning, with the reward model supplying the reward signal — the policy is nudged, generation by generation, toward outputs the reward model scores highly.

The results from InstructGPT were the proof this was worth doing: human evaluators preferred outputs from a 1.3-billion-parameter InstructGPT model over outputs from the 175-billion-parameter base GPT-3 model — a model with roughly 100 times fewer parameters, winning on preference because of alignment training, not scale. InstructGPT also showed measurable gains in truthfulness and reductions in toxic output, with minimal loss of general capability on standard benchmarks.

RLHF is not the only alignment technique in use today — direct preference optimization (DPO) and reinforcement fine-tuning (RFT) are alternatives covered in the fine-tuning course in this catalog — but RLHF's three-stage structure (demonstrate, rank, optimize against a learned reward) is the pattern nearly every later alignment technique responds to or builds on.

How inference actually happens

Once a model is trained, generating a response is a loop, not a single computation. A decoder-only model produces output autoregressively: it computes a probability distribution over its entire vocabulary for the next token, a decoding strategy picks one token from that distribution, that token is appended to the sequence, and the whole process repeats until a stop condition is reached.

How the next token gets picked from that distribution is a deliberate choice with real trade-offs:

Greedy search always picks the single most likely next token. It's simple and deterministic, and it works fine for short, factual outputs — but for longer generations it tends to fall into repetition, since it never risks a lower-probability token even when that would lead somewhere better.
Sampling (multinomial sampling) picks a token at random, weighted by the model's full probability distribution, so every token with nonzero probability has some chance of being chosen. This produces more varied, natural-sounding text, because natural human language doesn't consist of an unbroken chain of the single most predictable next word.
Beam search tracks several candidate sequences ("beams") simultaneously and keeps the ones with the highest overall probability, letting it favor a sequence that starts slightly less likely but leads to a better result overall. It suits input-grounded tasks like translation or captioning more than open-ended chat.

Controlling generation: temperature, top-k, top-p

Sampling on its own can be too unpredictable to be useful, so production systems constrain it with a few standard parameters:

Temperature rescales the model's probability distribution before sampling. Lower temperature sharpens the distribution, concentrating probability on the already-likely tokens; as temperature approaches zero, sampling converges toward greedy search. Higher temperature flattens the distribution, giving lower-probability tokens a better chance and producing more varied, sometimes less coherent, output.
Top-k sampling restricts the choice to only the k most probable next tokens, then samples among just those, discarding the long tail of implausible candidates. A fixed k is a blunt instrument, though — it can exclude reasonable options when the distribution is confident and narrow, and admit weak ones when the distribution is flat and uncertain.
Top-p (nucleus) sampling fixes that by choosing the smallest set of tokens whose cumulative probability exceeds a threshold p, rather than a fixed count — so the candidate pool automatically grows when the model is uncertain and shrinks when it's confident.

These parameters are usually combined, not used alone, and the right combination depends on the task: low temperature and a narrow candidate pool for factual extraction, higher temperature and a wider pool for creative writing.

Making inference fast

Serving an LLM at scale runs into a different set of problems than getting a single correct answer: memory, latency, and throughput.

KV caching. Naively, generating each new token would require recomputing attention over every previous token from scratch. A key-value (KV) cache stores the keys and values computed for earlier tokens so they don't need to be recomputed at every step — the standard optimization behind essentially all production LLM serving.
Speculative decoding. A small, fast assistant model proposes several candidate tokens ahead of the main model; the main model then verifies them in a single forward pass instead of generating them one at a time. When the assistant's guesses are correct, the main model gets those tokens "for free," with no loss of accuracy, since the larger model still determines what counts as correct.
Continuous batching. Real traffic doesn't arrive as one clean batch — requests come in at different times, at different lengths. Continuous batching groups incoming requests dynamically to keep GPU utilization and throughput high, rather than waiting to accumulate a fixed-size batch before starting.
Quantization. Storing model weights at lower numerical precision (8-bit or 4-bit instead of full 32-bit or 16-bit) cuts the memory needed to load a model, which matters directly: a 70-billion-parameter model needs roughly 256GB of memory at full precision, more than any single accelerator available today provides, but far less at reduced precision. The trade-off is a small amount of extra compute to quantize and dequantize weights, and a potential (usually small) quality cost.
Efficient attention implementations. Self-attention's compute and memory cost grows quadratically with sequence length, which is punishing for long-context requests. Techniques like FlashAttention restructure the computation to reduce memory reads and writes, speeding up inference without changing the model's outputs.

None of these techniques change what a model can compute — they change how fast and how cheaply it can compute it, which is a large part of what separates a research demo from a production system.

Where this course stops

This is a deeper look at two specific mechanics: how pretraining scale gets decided, how RLHF turns a base model into an assistant, and how inference actually runs, including the decoding and serving-optimization techniques that shape latency, cost, and output quality. It does not cover the transformer architecture those mechanics run on top of — see the Transformer Models course for that — or the adaptation techniques (fine-tuning, DPO, RFT) covered in the LLM Fine-Tuning course. It also doesn't walk through implementing any of these techniques in a specific framework; the links below go to the primary sources and official documentation for that.