LLM Foundations
LLM foundations covers the core concepts behind large language models: transformer architecture, tokenization, pre-training objectives, scaling laws, emergent capabilities, and the practical implications of how these models represent and generate language.
itArtificial intelligence and machine learning | OpenSkills.info
Course pathWalk it in order
Look it upDip in anytime
Go furtherLeaves this page
Don't Panic
Don't Panic — LLM Foundations
A large language model is a neural network with one relentlessly ordinary job: predict the next token. It reads pieces of text, turns them into numbers, and selects another piece. Repeat that at tremendous scale and it can write code, summarize a report, translate a sentence, or produce JSON. This is not a tiny librarian hiding inside a server. It is a very determined pattern completer with an impressive stationery budget.
The trick is the transformer, the architecture that lets each token pay attention to the other tokens in the input. Older language models moved through text in sequence and lost the plot over distance. A transformer can process the input in parallel, which is why a relationship near the start can still matter near the end. Its learned weights carry patterns from training; the prompt supplies the situation at hand. Confusing those two is how a request for current facts becomes an expensive confidence exercise.
Training also has a small bureaucratic arc. Pretraining makes a base model good at continuing text. Post-training, including supervised fine-tuning and reinforcement learning from human feedback, teaches it to behave more like an assistant. During inference, it generates one token at a time. Temperature changes how broadly it samples from possible next tokens. None of this gives it a secret fact-checker, persistent memory, or hands. It needs supplied context, stored state, or external tools for those jobs.
The surprise is that a large context window is still a budget, not a warehouse. Your instructions, history, added documents, and the reply all occupy it. More material can help, but it can also crowd out the answer you needed. The rest of an application matters for the same reason: system prompts shape behavior, tools provide actions and retrieval, memory preserves state, and guardrails check the result. The model is the engine; the surrounding parts are the brakes, map, and inconveniently necessary seat belts.
Read the Intro for the full architecture and training path. Use the Slides when the token-to-transformer-to-application chain needs a quick visual map. Keep the Cheatsheet nearby for parameters, limits, and the difference between a model generating text and a system doing useful work. Then follow the Reference links to move from the mental model to the documentation and papers that make its details less mysterious.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://learn.microsoft.com/en-us/agent-framework/journey/llm-fundamentals
Supports
- An LLM is a neural network trained on massive text to predict the next token in a sequence
- Two components model weights and architecture code (transformer)
- Tokens are sub-word units from a fixed vocabulary, roughly 3/4 of an English word
- Training stages pretraining (next-token prediction), SFT (instruction following), RLHF (human preferences), reasoning RL (chain of thought)
- Base model is a text-completion engine that does not follow instructions reliably
- Inference is autoregressive one token at a time with KV-cache optimization
- Context window is finite and holds input plus output
- Temperature controls randomness of token selection
- Even at temperature 0 LLMs are not fully deterministic
- LLMs cannot act on the world only generate text
- Tool calling is token generation the model outputs structured tokens representing a tool call
- Limitations no real-time knowledge, hallucination, no persistent memory, limited math and logic, non-determinism
- https://developers.google.com/machine-learning/crash-course/llm
Supports
- A language model estimates the probability of a token or sequence of tokens occurring within a longer sequence
- Tokens could be words, subwords, or single characters
- LLMs are advanced language models with vast parameters and datasets
- https://huggingface.co/learn/llm-course/chapter1/4
Supports
- The transformer architecture was introduced in June 2017
- Key models in the transformer lineage GPT (2018), BERT (2018), GPT-2 (2019), GPT-3 (2020), Llama (2023), Mistral (2023), Gemma (2024)
- Transformers replaced sequential processing with parallel self-attention
- https://arxiv.org/abs/1706.03762
Supports
- The transformer model is solely based on attention mechanisms without convolutional or recurrent layers
- Self-attention computes relationships between all positions in a sequence in parallel
- Multi-head attention runs multiple attention computations in parallel to capture different relationship types
- https://platform.openai.com/docs/models
Supports
- OpenAI provides models through its API
- https://docs.anthropic.com/en/docs/get-started
Supports
- Anthropic provides Claude through an API
- https://ai.google.dev/gemini-api/docs
Supports
- Google provides the Gemini API for generative AI applications
- https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html
Supports
- Amazon Bedrock provides access to foundation models through AWS
- https://learn.microsoft.com/en-us/azure/ai-foundry/what-is-azure-ai-foundry
Supports
- Azure AI Foundry provides tools for building AI applications
- https://huggingface.co/docs/inference-providers/index
Supports
- Hugging Face Inference Providers offers access to models from multiple providers
- https://openai.com/index/language-unsupervised/
Supports
- GPT introduced generative pretraining followed by task-specific fine-tuning
- https://arxiv.org/abs/1810.04805
Supports
- BERT introduced bidirectional transformer pretraining with masked-language and next-sentence objectives
- https://openai.com/index/better-language-models/
Supports
- GPT-2 reported larger-scale language-model generation
- https://arxiv.org/abs/2005.14165
Supports
- GPT-3 described few-shot task conditioning with a 175-billion-parameter autoregressive language model
- https://arxiv.org/abs/2203.02155
Supports
- InstructGPT described supervised fine-tuning and reinforcement learning from human feedback
- https://arxiv.org/abs/2302.13971
Supports
- LLaMA described a collection of foundation language models
- https://arxiv.org/abs/2303.08774
Supports
- GPT-4 accepts image and text inputs and produces text outputs
- https://arxiv.org/abs/2407.21783
Supports
- Llama 3.1 described multilingual models with 8B, 70B, and 405B parameter variants