Topic 1 · How a model works
LLM fundamentals
From raw text to a probability for every possible next token.
A language model never sees letters. Text is cut into tokens, each token becomes an integer, and each integer becomes a vector. Those vectors flow through a stack of layers, and at the end a single vector is turned into one score for every token in the vocabulary.
Generation is just that process on repeat: pick a token from the scores, append it, run again. Almost everything else in this guide, from KV caches to agent runtimes, exists to make that loop cheaper, faster or more useful.
In this topic
25
terms, each defined below
Text in, next token out
Type anything. The text is split into subword tokens, mapped to IDs and embeddings, and the model scores candidates for the next token. Press Generate to append the top pick and run the loop again. Token IDs and scores here are illustrative, not from a real tokenizer.
Tokens57 characters → 13 tokens
- The134736
- ·model97176
- ·reads49300
- ·one121
- ·token55416
- ·at53710
- ·a99398
- ·time6410
- ·and71140
- ·pred95888
- icts6820
- ·the61326
- ·next22282
Embedding of each token (first 12 of thousands of dimensions)
Next-token probabilities
- ·token55%
- ·word26%
- ·step11%
- ·one5%
- ·layer3%
- Text
- Tokens
- Token IDs
- Embeddings
- Transformer × N
- LM head + softmax
Glossary
25 terms
Tokens
The units a language model reads and writes. A token is usually a word, a piece of a word, a punctuation mark or a byte sequence. In English one token is on average about three quarters of a word.
Tokenization
Splitting raw text into tokens before the model sees it. Most LLMs use subword schemes such as byte-pair encoding (BPE), which keep frequent words whole and break rare words into reusable pieces.
Tokenizer
The component that converts text to tokens and back. It is built before the model is trained and ships with the weights; swapping it breaks the model, because every token ID would suddenly mean something else.
Token IDs
The integer index of each token in the vocabulary. A word like "Hello" might become 9906. The network only ever sees these numbers, never characters.
Vocabulary
The fixed list of every token the tokenizer can produce, including special tokens. Each entry has exactly one ID and one row in the embedding table.
Vocabulary size
How many entries the vocabulary has, typically 32K to 256K in current models. A larger vocabulary packs text into fewer tokens, especially code and non-English text, but grows the embedding table and the LM head.
Token embeddings
Learned vectors, one per vocabulary entry. Looking up a token ID returns its row of the embedding matrix, which turns a discrete symbol into numbers the network can compute with.
Embedding dimension
The length of each embedding vector, also called the hidden size or d_model, for example 4,096 or 7,168. It sets the width of the residual stream that runs through every layer.
Hidden states
The vectors carried between layers, one per token position. Each block reads them, adds an update and passes them on, so they gradually shift from "which token is this" to "what should come next".
Final hidden state
The hidden state at the last position after the final layer and final norm. It is the only vector the LM head needs to score the next token.
Positional embeddings
Information about where each token sits in the sequence. Attention by itself ignores order, so models either add learned position vectors to the embeddings or, more commonly today, rotate queries and keys by position (RoPE).
Autoregressive generation
Producing text one token at a time, where each new token is appended to the input and fed back in to predict the next. It is why generation is sequential and why output length drives latency.
Forward pass
One run of inputs through the network, from embeddings to logits. Training pairs every forward pass with a backward pass; inference only does forward passes, one per generated token after the prompt.
Model parameters
All the learned numbers in a network: embedding tables, attention and FFN weight matrices, norm scales. A "70B" model has roughly seventy billion of them.
Model weights
The stored values of those parameters, usually as files in bf16, fp8 or a quantized format. "Open weights" means these files can be downloaded, even if the training data and code cannot.
Weight matrices
The 2D arrays that hold most parameters: query, key, value and output projections, and the FFN up, gate and down projections. Nearly all compute in an LLM is multiplying activations by them.
Matrix multiplication
The core operation of every layer (GEMM): activations times weights. GPUs are built around doing it fast, and how quickly they can do it and feed it with data decides model speed.
LM head
The final linear layer that maps the final hidden state to one score per vocabulary token. Some models tie it to the embedding matrix to save parameters.
Output projection
A linear layer that maps a sublayer's result back to the hidden size, such as attention's W_O after the heads are concatenated. The LM head is sometimes called the output projection of the whole model.
Logits
The raw, unnormalised scores the LM head produces, one per vocabulary token. Higher means more likely, and they can be any real number.
Softmax
The function that turns scores into probabilities: exponentiate each score, then divide by the sum. It is used on the logits at the output and on attention scores inside every attention layer.
Probability distribution
The softmax output over the whole vocabulary: non-negative values that add up to 1. Decoding strategies decide how to pick one token from it.
Next-token prediction
The training objective of LLMs: given all previous tokens, predict the next one. Repeated over trillions of tokens, this one task is enough to learn grammar, facts and patterns of reasoning.
Next-token generation
Using a trained model to produce the next token at inference time: run a forward pass, read the distribution, pick a token with a decoding strategy, append it.
Detokenization
Turning generated token IDs back into text. Byte-level tokenizers can split one character (an emoji, say) across tokens, so streaming output is buffered until the bytes form valid text.