# 4. LLM inference internals

> Why the first token is slow and the rest are fast.

Track: Serving at scale. 22 terms.

Every request has two very different halves. Prefill reads the whole prompt in one big parallel pass and fills the KV cache. Decode then produces one token per step, and each step has to read all the model weights plus the cache for very little arithmetic.

That split explains most serving behaviour. Prefill is limited by compute and sets time to first token; decode is limited by memory bandwidth and sets streaming speed. Caching, speculative decoding and careful scheduling all attack one half or the other.

**Interactive figure on the web page: Prefill, then decode.** The prompt's tokens are processed together in one pass, filling the KV cache. After that, each step adds one token. Turn the cache off to see why it exists: without it every step recomputes the whole sequence, and the work per token keeps growing.

## Terms

### Inference

Running a trained model to get outputs. For LLMs it splits into processing the prompt, then generating tokens one by one, and the two halves behave very differently.

### Prefill phase

Processing all prompt tokens in one batched forward pass while computing and caching their keys and values. It is compute-bound, and its length mostly sets the time to first token.

### Decode phase

Generating output one token at a time. Each step is a forward pass over a single new token that reads the whole KV cache, which makes it memory-bandwidth-bound.

### Prompt processing

Another name for prefill, sometimes counting tokenization and chat-template formatting too. With long prompts it becomes the main cost of a request.

### Time to first token (TTFT)

The delay from sending a request to receiving the first output token: queueing plus prefill, whose final position already yields the first token. It is what users feel as responsiveness.

### Time per output token (TPOT)

The average time between output tokens after the first. Its inverse is the streaming speed of one request in tokens per second.

### Inter-token latency

The gap between two consecutive streamed tokens. TPOT is its average; its spikes, often caused by other requests' prefills sharing the GPU, are what make streaming stutter.

### KV cache

The stored keys and values of every earlier token in every layer, so each decode step only computes K and V for the newest token. It trades memory for compute and often fills most of a serving GPU.

See: [Grouped-Query Attention](https://llm-atlas-7or.pages.dev/concepts/gqa.md), [Multi-Head Latent Attention](https://llm-atlas-7or.pages.dev/concepts/mla.md), [Cross-Layer KV Sharing](https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md), [related](https://llm-atlas-7or.pages.dev/models/?sort=kv)

### KV-cache reuse

Using keys and values computed for one request in another that starts with the same tokens, which skips that part of prefill entirely.

### KV-cache allocation

How an engine hands out GPU memory for caches. Paged allocation, as in vLLM's PagedAttention, splits it into fixed-size blocks so little memory is reserved but left unused.

### KV-cache capacity

How many tokens of cache fit in GPU memory once the weights are loaded. It caps concurrent sequences times their length; compare the per-token cache figures in the gallery.

See: [related](https://llm-atlas-7or.pages.dev/models/?sort=kv)

### Prefix caching

Keeping cache blocks for common prompt beginnings (system prompts, few-shot examples, earlier turns) and matching new requests against them automatically.

### Prompt caching

The API feature built on prefix caching: a repeated prompt prefix is billed at a discount and answered faster. The cached part must usually be identical and at the start of the prompt.

### Cache eviction

Removing cache blocks when memory runs out, usually the least recently used prefixes first. An evicted prefix has to be recomputed if it is needed again.

### Cache preemption

Pausing a running sequence and freeing its cache, either by swapping it to CPU memory or by dropping it and recomputing later, so other requests can proceed when memory is full.

### Attention computation

The QKᵀ, softmax and weighted-sum work inside each attention layer. It grows with context length; fused kernels like FlashAttention compute it without ever storing the full score matrix.

See: [Sliding Window Attention](https://llm-atlas-7or.pages.dev/concepts/swa.md), [DeepSeek Sparse Attention](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md)

### Compute-bound workloads

Work whose speed is capped by arithmetic throughput rather than memory. Prefill and large-batch matrix multiplications are compute-bound.

### Memory-bandwidth-bound workloads

Work capped by how fast data can stream from memory. Small-batch decode is the classic case: each token needs every weight and the whole cache to be read for very little maths.

### Speculative decoding

A fast drafter proposes several tokens, and the large model checks them all in one forward pass, keeping the longest accepted run. The output distribution is unchanged, but each expensive pass can yield several tokens.

See: [Multi-Token Prediction](https://llm-atlas-7or.pages.dev/concepts/mtp.md)

### Draft model

The cheap proposer in speculative decoding: a smaller model from the same family, extra prediction heads, a multi-token-prediction module, or even n-gram lookups from the prompt.

See: [Multi-Token Prediction](https://llm-atlas-7or.pages.dev/concepts/mtp.md)

### Token acceptance

The verification step of speculative decoding, and the rate at which drafted tokens are accepted. Higher acceptance means more tokens per expensive forward pass.

### Model execution overhead

Time spent outside the core maths: scheduler logic, kernel launches, sampling and CPU-GPU synchronisation. At small batch sizes it can eat a large share of each step; CUDA graphs reduce it.

Web page: https://llm-atlas-7or.pages.dev/learn/inference/

_Definitions written for LLM Atlas._