Topic 2 · How a model works
Transformer architecture
The repeated block that almost every modern LLM is built from.
A decoder-only transformer is a tall stack of identical blocks. Each block has two sublayers: attention, where tokens look at each other, and a feed-forward network, where each token is processed on its own. Both add their result back onto a shared residual stream.
The ideas in the Concepts section, such as GQA, MLA, sliding windows and MoE, are mostly changes to one of these two sublayers. Once the basic block makes sense, every architecture diagram in the gallery reads as a set of edits to it.
Who attends to whom
Each row is a query token, each column a key it can look at. Pick a head to see a different learned pattern, and toggle the causal mask to see why a decoder can only look backwards. Hover or tap a row to read its weights. Patterns are illustrative.
| The | cat | chased | the | mouse | because | it | was | |
|---|---|---|---|---|---|---|---|---|
Query “it” attends to
- cat65%
- mouse16%
- it6%
- The3%
- chased3%
The output for this token is the weighted average of those tokens’ value vectors.
One transformer block (pre-norm)
- RMSNorm
- Attention
- + residual add (input skips around)
- RMSNorm
- FFN
- + residual add (input skips around)
Glossary
20 terms
Decoder-only Transformer
The architecture of nearly every current LLM: a stack of transformer blocks with causal self-attention and no separate encoder. It reads a sequence and predicts the next token at every position.
Transformer blocks
The repeated unit of the network, often just called layers. A modern pre-norm block runs RMSNorm, attention, a residual add, RMSNorm, FFN and another residual add. Models stack anywhere from about 24 to over 100 of them.
Self-attention
The operation that lets each token gather information from other tokens in the same sequence. Every position builds a query, compares it with every key, and takes a weighted average of the values.
Attention layers
The attention sublayer of each block: Q, K and V projections, the attention computation and the output projection. Variants such as GQA, MLA and sliding-window attention change this layer to save memory or compute.
Query (Q)
A vector each token produces that asks "what am I looking for?". Its dot product with every key decides how much that token attends to each position.
Key (K)
A vector each token produces that advertises "what I contain". During generation the keys of past tokens are stored in the KV cache.
Value (V)
The vector a token hands over when it is attended to: the information that actually moves. The attention output is a weighted sum of values.
Attention masks
A matrix added to the attention scores that blocks certain pairs, usually by setting them to minus infinity before the softmax. Used for causality, padding and packing several documents into one sequence.
Causal attention
Attention where a token may only look at itself and earlier tokens, enforced with an upper-triangular mask. It lets the model learn from every position of a sequence in one pass without seeing the answer.
Multi-head attention
Several attention operations (heads) run in parallel, each with its own Q, K and V projections over a slice of the hidden size, then their results are concatenated. Different heads learn different patterns.
Rotary Positional Embeddings (RoPE)
Encodes position by rotating pairs of query and key dimensions by an angle that grows with the token's position. The attention score then depends on relative distance, and the rotation frequencies can be rescaled to extend the context.
RMSNorm
A normalisation that divides each vector by its root-mean-square and multiplies by a learned scale. It is cheaper than LayerNorm (no mean, no bias) and is the default in modern LLMs.
Layer normalization
Normalises each token's vector to zero mean and unit variance, then applies a learned scale and shift. Used in the original Transformer and GPT-2; most new models use RMSNorm instead.
Feed-forward network (FFN)
The per-token sublayer after attention: widen the vector (often 3 to 4 times), apply a nonlinearity, project back. In modern models it is a gated SwiGLU and holds most of the parameters.
Multi-layer perceptron (MLP)
Another name for the FFN sublayer: a small fully connected network applied to each token independently. In MoE models every expert is one MLP.
Residual connections
Skip paths that add each sublayer's input to its output, so layers write updates onto a shared residual stream instead of replacing it. They keep signals and gradients flowing through very deep stacks.
Residual addition
The add operation x + f(x) at the end of each sublayer. It is cheap, and it is the reason every layer must keep the same hidden size.
Final normalization
One last norm on the hidden states after the final block and before the LM head, which keeps the scale of the residual stream stable for the output layer.
Linear layers
Layers that multiply their input by a weight matrix, optionally adding a bias. Q, K, V and O projections, FFN projections, MoE routers and the LM head are all linear layers; modern LLMs mostly drop the biases.
Model architecture
The full specification of a network: layer count, hidden size, heads, attention type, FFN or MoE layout, norms, positions, vocabulary and context length. The gallery compares these choices across 100+ releases.