Topic 15 · Training, tools & research
Model training & post-training
How a pile of text becomes a model that follows instructions and uses tools.
Training is gradient descent at enormous scale. The model makes a prediction, a loss function measures how wrong it was, backpropagation works out how every parameter contributed, and an optimizer nudges them all to do slightly better. Repeat a few million times.
Pretraining on trillions of tokens builds knowledge and fluency. Post-training then shapes behaviour: supervised fine-tuning on demonstrations teaches the chat format, and preference and reinforcement learning teach the model to be helpful, follow rules and use tools well.
In this topic
17
terms, each defined below
Gradient descent, by hand
A ball rolling downhill on a loss surface. Each step follows the negative gradient, scaled by the learning rate. Too small and progress crawls; too large and it overshoots and zigzags. Momentum, the idea behind Adam's first moment, smooths the path. Drag the learning rate and press Train.
Loss per stepfinal 0.002
A reasonable rate: fast across the steep direction, slower along the valley.
- stage 1
Pretraining
Next-token prediction on trillions of tokens. Knowledge and fluency.
- stage 2
SFT
Imitate curated demonstrations. Learns the chat format.
- stage 3
Preference & RL
Reward helpful, correct, safe answers and good tool use.
- stage 4
Evaluation
Benchmarks, safety tests and regression checks before release.
Glossary
17 terms
Pretraining
The first and largest training stage: next-token prediction on trillions of tokens of web text, code and books. It gives the model its knowledge and language ability.
Supervised fine-tuning (SFT)
Training on curated prompt and response pairs so a base model learns to follow instructions and a chat format.
Post-training
Everything after pretraining that shapes behaviour: SFT, preference optimisation (RLHF, DPO), reinforcement learning with verifiable rewards and safety training.
Tool-use training
Fine-tuning on trajectories that contain tool calls and results, and reinforcement learning in tool environments, so models learn when and how to call tools.
Training examples
The individual inputs and targets, or whole trajectories, a model learns from. In post-training, quality and diversity matter more than sheer count.
Learning from demonstrations
Training a model to imitate example behaviour shown by people or stronger models (imitation learning). SFT is a form of it.
Gradient descent
The optimisation method behind all of it: compute how the loss changes with each parameter (the gradient) and move every parameter a small step in the opposite direction.
Backpropagation
The algorithm that computes the gradient for every parameter efficiently, by applying the chain rule backwards through the network.
Loss functions
Functions that measure how wrong a prediction is. LLM pretraining uses cross-entropy on the next token.
Optimizers
Algorithms that turn gradients into parameter updates, adding momentum, adaptive step sizes and weight decay, such as SGD, AdamW and Muon.
Adam optimizer
An optimiser that keeps running averages of each parameter's gradient and squared gradient, giving every parameter its own adaptive step size. AdamW, with decoupled weight decay, is the standard for LLMs.
Training datasets
The text, code and trajectories used for training. Filtering, deduplication and mixing ratios shape the result as much as size does.
Model behavior
How a model actually responds, including style, helpfulness, refusals and tool use. It is shaped mainly in post-training.
Model capabilities
What a model can do, such as reasoning, coding, languages, long context and tool use. Mostly built in pretraining and drawn out in post-training.
Training feedback
The signals used to improve a model: human preferences, reward models, verifiers or the results of running code.
Model evaluation
Measuring a model's quality on held-out tasks, safety tests and real-world use.
Benchmark evaluation
Scoring a model on standard benchmarks such as MMLU, GPQA, SWE-bench and MATH-500. Comparable across models, but contamination and overfitting limit what a score means.