Topic 3 · How a model works
Decoding & sampling
How a probability distribution becomes an actual word.
The model hands back a probability for every token in its vocabulary. Decoding is the policy that picks one. Always taking the top token gives repeatable but often flat text; sampling adds variety, and settings like temperature, top-k and top-p control how much.
The same step is where structure is enforced. By masking every token that would break a grammar or JSON schema, an engine can guarantee valid output instead of hoping for it. Tool calls in agents depend on exactly this.
In this topic
19
terms, each defined below
Sampling playground
Candidates for the word after "The weather today is". Move the sliders and watch the distribution reshape: temperature sharpens or flattens it, top-k and top-p cut the tail. Sample a few times to see how often each word wins.
The weather today is …
- sunny37%
- cloudy22%
- warm15%
- rainy11%
- perfect6%
- cold5%
- strange2%
- purple0.9%
- banana0.4%
- seventeen0.2%
Solid bar: probability after filtering. Dashed outline: after temperature only. Right column: times drawn.
10 of 10 candidates can be picked.
Glossary
19 terms
Decoding
The strategy that turns the model's probability distribution into an actual token at each step, repeated until a stopping condition is met.
Sampling
Picking the next token at random in proportion to its probability, instead of always taking the most likely one. It gives varied, more natural text at the cost of determinism.
Sampling settings
The knobs exposed by an API or engine: temperature, top-k, top-p, min-p, penalties, seed and maximum tokens. Code and data extraction usually want little randomness; brainstorming wants more.
Temperature
A number the logits are divided by before the softmax. Below 1 sharpens the distribution toward the top token, above 1 flattens it, and 0 is treated as greedy decoding.
Temperature scaling
The operation logits ÷ T itself. It changes how far apart the token probabilities are without changing their order.
Top-k sampling
Keep only the k most likely tokens, renormalise and sample among them. It cuts off the long tail of unlikely tokens, but uses the same k whether the model is confident or unsure.
Top-p / nucleus sampling
Keep the smallest set of top tokens whose probabilities add up to p (say 0.9), then sample. The set shrinks when the model is confident and grows when it is unsure.
Greedy decoding
Always choose the single most likely token. Deterministic and cheap, but prone to repetition and bland text in open-ended writing.
Argmax
The operation that returns the position of the largest value. Greedy decoding is an argmax over the logits at every step.
Repetition penalty
Lowers the scores of tokens that already appear in the context, for example by dividing their positive logits by a factor, to discourage loops.
Frequency penalty
Subtracts an amount proportional to how often a token has already been generated, so heavily repeated tokens are pushed down more. The related presence penalty applies a flat amount once a token has appeared at all.
Logit processing
Any step that changes the logits between the LM head and the final pick: temperature, penalties, logit bias, masks. Engines run them as a chain of logit processors.
Logit masking
Setting some logits to minus infinity so those tokens can never be chosen. It is the mechanism behind constrained decoding and banned tokens.
Constrained decoding
Limiting generation so the output always matches a formal rule, such as a regex, a grammar or a fixed list of options, by masking every token that would break it at each step.
Structured generation
Producing output in a guaranteed shape, such as JSON, XML or a function call, rather than asking nicely. Usually built on constrained decoding driven by a grammar or schema.
JSON-constrained decoding
Constrained decoding whose grammar is compiled from a JSON Schema, so the output is always parseable JSON with the required fields. Used for tool calls and data extraction.
Token selection
The last step of each decoding iteration: choose one token ID from the processed distribution, by argmax or by sampling.
Stopping conditions
The rules that end generation: an end-of-sequence token, a stop string, the maximum token count, or the end of a tool call.
End-of-sequence token
A special token such as <|endoftext|> or <|im_end|> that the model learns to emit when its answer is complete. Generation stops when it is picked.