LLM Foundations
LLM foundations covers the core concepts behind large language models: transformer architecture, tokenization, pre-training objectives, scaling laws, emergent capabilities, and the practical implications of how these models represent and generate language.
itArtificial intelligence and machine learning | OpenSkills.info
Intro
LLM Foundations
A large language model is a neural network trained on massive amounts of text to predict the next token in a sequence. Through that single training objective — given everything before this point, what comes next? — the model learns language structure, factual knowledge, reasoning patterns, and the ability to follow instructions.
That description is deceptively simple. The result is a system that can generate code, translate languages, summarize documents, answer questions, and produce structured data on demand. Understanding how it works, what it can do, and where it fails gives you the foundation to use it well.
What an LLM actually is
Strip away the marketing and an LLM is two things:
- Model weights — billions of numerical parameters learned during training. These encode everything the model "knows."
- Architecture code — the neural network structure (a transformer) that runs those weights to produce output.
Continue the course
This section is part of the paid course.
See pricing to subscribe, or log in if you already have access.
Where this skill leads
Relevant careers
See how this topic contributes to broader role-level skill maps.
Sources
- https://learn.microsoft.com/en-us/agent-framework/journey/llm-fundamentals
Supports
- An LLM is a neural network trained on massive text to predict the next token in a sequence
- Two components model weights and architecture code (transformer)
- Tokens are sub-word units from a fixed vocabulary, roughly 3/4 of an English word
- Training stages pretraining (next-token prediction), SFT (instruction following), RLHF (human preferences), reasoning RL (chain of thought)
- Base model is a text-completion engine that does not follow instructions reliably
- Inference is autoregressive one token at a time with KV-cache optimization
- Context window is finite and holds input plus output
- Temperature controls randomness of token selection
- Even at temperature 0 LLMs are not fully deterministic
- LLMs cannot act on the world only generate text
- Tool calling is token generation the model outputs structured tokens representing a tool call
- Limitations no real-time knowledge, hallucination, no persistent memory, limited math and logic, non-determinism
- https://developers.google.com/machine-learning/crash-course/llm
Supports
- A language model estimates the probability of a token or sequence of tokens occurring within a longer sequence
- Tokens could be words, subwords, or single characters
- LLMs are advanced language models with vast parameters and datasets
- https://huggingface.co/learn/llm-course/chapter1/4
Supports
- The transformer architecture was introduced in June 2017
- Key models in the transformer lineage GPT (2018), BERT (2018), GPT-2 (2019), GPT-3 (2020), Llama (2023), Mistral (2023), Gemma (2024)
- Transformers replaced sequential processing with parallel self-attention
- https://arxiv.org/abs/1706.03762
Supports
- The transformer model is solely based on attention mechanisms without convolutional or recurrent layers
- Self-attention computes relationships between all positions in a sequence in parallel
- Multi-head attention runs multiple attention computations in parallel to capture different relationship types