Large Language Model Training and Inference
LLM training and inference covers the infrastructure, techniques, and engineering required to train large language models on massive text corpora and serve their predictions efficiently. It addresses distributed training, hardware selection, optimization, quantization, and serving architectures.
itArtificial intelligence and machine learning | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — Large Language Model Training and Inference
Large language models look like magic from the outside, but they are engineering projects with known constraints and tunable knobs. This course covers two specific mechanics: how the size of a training run gets decided, and what actually happens every time a model generates a response.
The first thing to know is that parameter count alone does not tell you what a model can do. For years, the assumption was to make models as large as possible. DeepMind's Chinchilla study proved that wrong: a 70-billion-parameter model trained on 1.4 trillion tokens beat a 280-billion-parameter model trained on a quarter of that data, using the same compute. The lesson is that model size and training data need to scale together, and a smaller model trained well can outperform a much larger one trained badly.
The second thing is that a freshly pretrained model is a text-completion engine, not an assistant. It predicts plausible next words; it does not inherently follow instructions or decline unsafe requests. Turning it into something useful requires a separate alignment step — RLHF, or reinforcement learning from human feedback — where human preference rankings train a reward model that guides the base model toward helpful, safe behavior. A 1.3-billion-parameter aligned model can beat a 175-billion-parameter unaligned one on human preference. Alignment is not a hack; it is a fundamentally different kind of training.
The third thing is that inference is a loop, not a single computation. Each token is generated one at a time, with a decoding strategy picking which token to append next. Temperature controls how creative or conservative that choice is. Top-p sampling adapts the pool of candidate tokens to the model's confidence. And a handful of engineering tricks — KV caching, speculative decoding, quantization — determine whether that loop runs at a cost you can actually afford.
If you remember nothing else: the gap between a research demo and a production system is almost never about model quality. It is about serving efficiency. Read the Cheatsheet for the parameter names, the Intro for the full explanation, and the Field Notes for the mistakes teams actually make.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://deepmind.google/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/
Supports
- Model size and training data should be scaled equally for optimal performance within a fixed compute budget
- Prior large models, including Gopher, were far too large for their compute budget and not trained on enough data
- A 4x smaller model trained on 4x more data would have been preferable for the compute used to train Gopher
- Chinchilla (70B parameters, 1.3-1.4T tokens) outperforms Gopher (280B parameters, ~300B tokens) on nearly every measured task at matched training compute
- https://arxiv.org/abs/2203.02155
Supports
- Three-stage RLHF process -- supervised fine-tuning on labeler demonstrations, reward model training on ranked outputs, reinforcement learning using the reward model
- The reinforcement learning stage uses proximal policy optimization (PPO) on the trained reward model
- Outputs from a 1.3B parameter InstructGPT model were preferred over outputs from the 175B parameter base GPT-3 model
- InstructGPT showed improvements in truthfulness and reductions in toxic output generation with minimal performance regressions on public NLP datasets
- https://huggingface.co/docs/transformers/main/en/generation_strategies
Supports
- Greedy search selects the next most likely token at each step and is the default decoding strategy
- Greedy search works well for short outputs but breaks down for longer sequences due to repetition
- Sampling (multinomial sampling) randomly selects a token based on the full probability distribution over the vocabulary
- Sampling reduces repetition and produces more creative and diverse outputs than greedy search
- Beam search tracks several candidate sequences and selects the one with the highest overall probability, suited to input-grounded tasks like translation or speech recognition
- https://huggingface.co/blog/how-to-generate
Supports
- Temperature adjusts the sharpness of the probability distribution; lowering it makes high-probability tokens more likely and approaches greedy decoding as it nears zero
- Top-k sampling filters to the k most probable next tokens and redistributes probability among them; a fixed k can exclude viable words in sharp distributions or include poor choices in flat distributions
- Top-p (nucleus) sampling chooses the smallest set of tokens whose cumulative probability exceeds a threshold p, adapting pool size to model confidence
- High-quality human language does not follow a distribution of purely high-probability next words, which is why greedy selection produces repetitive output
- https://huggingface.co/docs/transformers/main/en/llm_optims
Supports
- Loading a 70B parameter model requires roughly 256GB of memory for full precision weights, more than current GPUs (A100/H100, 80GB) provide
- A key-value (kv) cache stores past keys and values instead of recomputing them at each generation step
- Speculative decoding uses a smaller assistant model to generate candidate tokens verified by the larger model in a single forward pass, with no degradation in accuracy
- The assistant model should share the same tokenizer as the main model and be much smaller for the largest speedup
- Quantization reduces model weight precision (e.g., 8-bit or 4-bit) to lower memory usage, at the cost of a small amount of extra compute to quantize/dequantize
- Continuous batching groups incoming inference requests to maximize GPU utilization and throughput
- Self-attention compute and memory cost grows quadratically with input token count; FlashAttention-2 and SDPA reduce memory read/write operations to speed up inference
- https://arxiv.org/abs/2203.15556
Supports
- Chinchilla (70B parameters, 1.4T tokens) outperforms Gopher (280B parameters, ~300B tokens) at the same compute budget
- Model size and training tokens should scale roughly equally for compute-optimal training
- https://arxiv.org/abs/2205.14135
Supports
- FlashAttention reduces memory reads and writes by restructuring the attention computation, achieving wall-clock speedup without approximation
- IO-aware tiling reduces HBM accesses from quadratic to linear in sequence length for the attention computation
- https://arxiv.org/abs/2302.13971
Supports
- LLaMA models (7B to 65B) trained on public data achieve competitive performance with proprietary models
- Chinchilla-optimal training on public data can produce strong foundation models
- https://arxiv.org/abs/2307.09288
Supports
- Llama 2 uses RLHF alignment on open-weight models
- Llama 2 trains on 40% more tokens than LLaMA 1
- https://arxiv.org/abs/2401.04088
Supports
- Mixtral 8x7B uses 46.7B total parameters with 12.9B active per token via sparse mixture-of-experts
- Sparse MoE delivers frontier-level quality at reduced inference cost
- https://arxiv.org/abs/2407.21783
Supports
- Llama 3 trains on over 15 trillion tokens
- Continued scaling of training data reliably improves open-weight model quality
- https://github.com/microsoft/DeepSpeed
Supports
- ZeRO memory optimization shards optimizer states, gradients, and parameters across GPUs
- https://github.com/NVIDIA/Megatron-LM
Supports
- Tensor, pipeline, and sequence parallelism for multi-GPU training at scale
- https://docs.vllm.ai/
Supports
- PagedAttention manages GPU memory for LLM serving by paging KV cache blocks
- https://huggingface.co/docs/text-generation-inference/
Supports
- Hugging Face TGI provides continuous batching and tensor parallelism for production LLM serving
- https://nvidia.github.io/TensorRT-LLM/
Supports
- TensorRT-LLM optimizes LLM inference with kernel fusion and in-flight batching on NVIDIA GPUs
- https://github.com/TimDettmers/bitsandbytes
Supports
- 8-bit and 4-bit quantization for memory-efficient LLM inference and fine-tuning
- https://github.com/ggerganov/llama.cpp
Supports
- LLM inference in C/C++ with quantization for CPU and consumer hardware
- https://ollama.com/
Supports
- Simplified local LLM inference with a Docker-like interface
- https://github.com/axolotl-ai-cloud/axolotl
Supports
- Streamlined fine-tuning framework wrapping multiple training backends
- https://modal.com/
Supports
- Serverless GPU compute for LLM training and inference without infrastructure management
- https://www.together.ai/
Supports
- Managed inference API for open-weight and proprietary LLMs
- https://www.anyscale.com/
Supports
- Managed Ray-based platform for distributed LLM training and serving
- https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
Supports
- GPT-1 demonstrates transfer learning with a pretrained decoder-only transformer
- https://arxiv.org/abs/2005.14165
Supports
- GPT-3 at 175B parameters demonstrates strong few-shot learning without task-specific fine-tuning
- https://arxiv.org/abs/2101.03961
Supports
- Switch Transformer uses conditional computation to decouple model capacity from per-token compute
- https://arxiv.org/abs/2212.08073
Supports
- Reward hacking occurs when policies learn output patterns that score well on the reward model but are not genuinely preferred by humans