# 15. Model training & post-training

> How a pile of text becomes a model that follows instructions and uses tools.

Track: Training, tools & research. 17 terms.

Training is gradient descent at enormous scale. The model makes a prediction, a loss function measures how wrong it was, backpropagation works out how every parameter contributed, and an optimizer nudges them all to do slightly better. Repeat a few million times.

Pretraining on trillions of tokens builds knowledge and fluency. Post-training then shapes behaviour: supervised fine-tuning on demonstrations teaches the chat format, and preference and reinforcement learning teach the model to be helpful, follow rules and use tools well.

**Interactive figure on the web page: Gradient descent, by hand.** A ball rolling downhill on a loss surface. Each step follows the negative gradient, scaled by the learning rate. Too small and progress crawls; too large and it overshoots and zigzags. Momentum, the idea behind Adam's first moment, smooths the path. Drag the learning rate and press Train.

## Terms

### Pretraining

The first and largest training stage: next-token prediction on trillions of tokens of web text, code and books. It gives the model its knowledge and language ability.

See: [related](https://llm-atlas-7or.pages.dev/learn/fundamentals/#next-token-prediction)

### Supervised fine-tuning (SFT)

Training on curated prompt and response pairs so a base model learns to follow instructions and a chat format.

### Post-training

Everything after pretraining that shapes behaviour: SFT, preference optimisation (RLHF, DPO), reinforcement learning with verifiable rewards and safety training.

### Tool-use training

Fine-tuning on trajectories that contain tool calls and results, and reinforcement learning in tool environments, so models learn when and how to call tools.

See: [related](https://llm-atlas-7or.pages.dev/learn/tool-calling/)

### Training examples

The individual inputs and targets, or whole trajectories, a model learns from. In post-training, quality and diversity matter more than sheer count.

### Learning from demonstrations

Training a model to imitate example behaviour shown by people or stronger models (imitation learning). SFT is a form of it.

### Gradient descent

The optimisation method behind all of it: compute how the loss changes with each parameter (the gradient) and move every parameter a small step in the opposite direction.

### Backpropagation

The algorithm that computes the gradient for every parameter efficiently, by applying the chain rule backwards through the network.

### Loss functions

Functions that measure how wrong a prediction is. LLM pretraining uses cross-entropy on the next token.

### Optimizers

Algorithms that turn gradients into parameter updates, adding momentum, adaptive step sizes and weight decay, such as SGD, AdamW and Muon.

### Adam optimizer

An optimiser that keeps running averages of each parameter's gradient and squared gradient, giving every parameter its own adaptive step size. AdamW, with decoupled weight decay, is the standard for LLMs.

### Training datasets

The text, code and trajectories used for training. Filtering, deduplication and mixing ratios shape the result as much as size does.

### Model behavior

How a model actually responds, including style, helpfulness, refusals and tool use. It is shaped mainly in post-training.

### Model capabilities

What a model can do, such as reasoning, coding, languages, long context and tool use. Mostly built in pretraining and drawn out in post-training.

### Training feedback

The signals used to improve a model: human preferences, reward models, verifiers or the results of running code.

### Model evaluation

Measuring a model's quality on held-out tasks, safety tests and real-world use.

### Benchmark evaluation

Scoring a model on standard benchmarks such as MMLU, GPQA, SWE-bench and MATH-500. Comparable across models, but contamination and overfitting limit what a score means.

Web page: https://llm-atlas-7or.pages.dev/learn/training/

_Definitions written for LLM Atlas._