Topic 4 · Serving at scale
LLM inference internals
Why the first token is slow and the rest are fast.
Every request has two very different halves. Prefill reads the whole prompt in one big parallel pass and fills the KV cache. Decode then produces one token per step, and each step has to read all the model weights plus the cache for very little arithmetic.
That split explains most serving behaviour. Prefill is limited by compute and sets time to first token; decode is limited by memory bandwidth and sets streaming speed. Caching, speculative decoding and careful scheduling all attack one half or the other.
In this topic
22
terms, each defined below
Prefill, then decode
The prompt's tokens are processed together in one pass, filling the KV cache. After that, each step adds one token. Turn the cache off to see why it exists: without it every step recomputes the whole sequence, and the work per token keeps growing.
Sequence
KV cache (one slot per token, every layer)0 slots
Each new token only computes its own key and value, then reads the rest from the cache.
Tokens processed per forward pass
prefilldecode steps →
Total work so far: 0 token passes
Glossary
22 terms
Inference
Running a trained model to get outputs. For LLMs it splits into processing the prompt, then generating tokens one by one, and the two halves behave very differently.
Prefill phase
Processing all prompt tokens in one batched forward pass while computing and caching their keys and values. It is compute-bound, and its length mostly sets the time to first token.
Decode phase
Generating output one token at a time. Each step is a forward pass over a single new token that reads the whole KV cache, which makes it memory-bandwidth-bound.
Prompt processing
Another name for prefill, sometimes counting tokenization and chat-template formatting too. With long prompts it becomes the main cost of a request.
Time to first token (TTFT)
The delay from sending a request to receiving the first output token: queueing plus prefill, whose final position already yields the first token. It is what users feel as responsiveness.
Time per output token (TPOT)
The average time between output tokens after the first. Its inverse is the streaming speed of one request in tokens per second.
Inter-token latency
The gap between two consecutive streamed tokens. TPOT is its average; its spikes, often caused by other requests' prefills sharing the GPU, are what make streaming stutter.
KV cache
The stored keys and values of every earlier token in every layer, so each decode step only computes K and V for the newest token. It trades memory for compute and often fills most of a serving GPU.
KV-cache reuse
Using keys and values computed for one request in another that starts with the same tokens, which skips that part of prefill entirely.
KV-cache allocation
How an engine hands out GPU memory for caches. Paged allocation, as in vLLM's PagedAttention, splits it into fixed-size blocks so little memory is reserved but left unused.
KV-cache capacity
How many tokens of cache fit in GPU memory once the weights are loaded. It caps concurrent sequences times their length; compare the per-token cache figures in the gallery.
Prefix caching
Keeping cache blocks for common prompt beginnings (system prompts, few-shot examples, earlier turns) and matching new requests against them automatically.
Prompt caching
The API feature built on prefix caching: a repeated prompt prefix is billed at a discount and answered faster. The cached part must usually be identical and at the start of the prompt.
Cache eviction
Removing cache blocks when memory runs out, usually the least recently used prefixes first. An evicted prefix has to be recomputed if it is needed again.
Cache preemption
Pausing a running sequence and freeing its cache, either by swapping it to CPU memory or by dropping it and recomputing later, so other requests can proceed when memory is full.
Attention computation
The QKᵀ, softmax and weighted-sum work inside each attention layer. It grows with context length; fused kernels like FlashAttention compute it without ever storing the full score matrix.
Compute-bound workloads
Work whose speed is capped by arithmetic throughput rather than memory. Prefill and large-batch matrix multiplications are compute-bound.
Memory-bandwidth-bound workloads
Work capped by how fast data can stream from memory. Small-batch decode is the classic case: each token needs every weight and the whole cache to be read for very little maths.
Speculative decoding
A fast drafter proposes several tokens, and the large model checks them all in one forward pass, keeping the longest accepted run. The output distribution is unchanged, but each expensive pass can yield several tokens.
Draft model
The cheap proposer in speculative decoding: a smaller model from the same family, extra prediction heads, a multi-token-prediction module, or even n-gram lookups from the prompt.
Token acceptance
The verification step of speculative decoding, and the rate at which drafted tokens are accepted. Higher acceptance means more tokens per expensive forward pass.
Model execution overhead
Time spent outside the core maths: scheduler logic, kernel launches, sampling and CPU-GPU synchronisation. At small batch sizes it can eat a large share of each step; CUDA graphs reduce it.