Topic 5 · Serving at scale
Batching & scheduling
Sharing one GPU between many conversations without making anyone wait too long.
Reading a model's weights from memory is the expensive part of decoding, and the cost is the same whether those weights are used for one request or sixty-four. Batching serves many requests per read, which is why one GPU can stream to dozens of users at once.
The hard part is scheduling. Requests arrive at random and finish at different lengths. Continuous batching refills a slot the moment a sequence finishes, and the scheduler's policy decides who runs, who waits and how the tail latency looks.
In this topic
24
terms, each defined below
Static vs continuous batching
Eight requests of different lengths share four GPU slots. With static batching the whole batch waits for its longest member, leaving hatched slots idle. Continuous batching starts the next request the moment a slot frees up. Watch the utilisation and finish times.
time step 0 / 15
Static batching
0/8 done · utilisation 60% · avg finish t=8.5
Continuous batching
0/8 done · utilisation 75% · avg finish t=6.5
Same eight requests (lengths 6, 2, 8, 3, 5, 2, 7, 3 steps), same four slots. Hatched cells are slots held by a finished request while its batch waits.
Glossary
24 terms
Batching
Processing several requests in the same forward pass. Weights are read from memory once and reused for every request in the batch, the main lever for serving throughput.
Sequence batching
Grouping multiple token sequences into one batch, by padding them to a common length or packing them end to end.
Static batching
Collect a fixed batch, run it to completion, then start the next. Simple, but short requests wait for the longest one and finished slots sit empty.
Dynamic batching
Forming batches on the fly from whatever requests arrive within a short time window or up to a size limit, trading a little waiting for better utilisation.
Continuous batching
Rebuilding the batch at every decode step: finished sequences leave immediately and waiting requests join. Also called in-flight or iteration-level batching; it is standard in vLLM, SGLang and TensorRT-LLM.
Batch size
How many sequences, or tokens, are processed together in one step. Bigger batches raise throughput until compute saturates, but slow each request and need more KV-cache memory.
Batched forward pass
One forward pass over the whole batch, with tokens from different requests stacked so each weight matrix is multiplied once for all of them.
Batch scheduling
Deciding at every step which requests run, how many prompt tokens to mix in with decode tokens, and who waits.
Request admission
Deciding whether a newly arrived request can start now, given free cache memory and the batch budget, or must wait in the queue.
Inference scheduler
The engine component that makes batching, admission and preemption decisions every iteration. Its policy shapes throughput, fairness and tail latency.
Request queue
Requests that have arrived but are not yet running. Its depth is an early warning of overload.
Queueing delay
Time a request waits before its prefill starts. Under load it can be far larger than the compute time, and it is often the hidden part of TTFT.
Concurrency
The number of requests in flight at once, running or waiting. Throughput rises with concurrency until the GPU or its memory saturates.
Concurrent inference
Serving many independent requests on the same model instance at the same time through batching, rather than one after another.
Active sequences
Sequences in the running batch with their KV cache resident on the GPU. Their number is bounded by memory and the engine's maximum batch setting.
Sequence completion
A sequence reaching a stopping condition. With continuous batching its slot and cache blocks are handed to the next request right away.
Dynamic batch membership
The set of sequences in a batch changing from step to step. Continuous batching relies on it, and it requires per-sequence cache bookkeeping.
Arithmetic intensity
Floating-point operations done per byte read from memory. Batch-1 decode does about one per byte; batching raises it roughly in proportion to batch size until the hardware becomes compute-bound.
Weight reuse
Using each weight loaded from memory for many tokens. That reuse is exactly what batching buys.
GPU utilization
How busy the GPU is. A busy reading in nvidia-smi only means some kernel was running, not that the tensor cores were saturated; model FLOPs utilisation (MFU) is the stricter measure.
Scheduling policy
The rule for ordering work: first come first served, shortest job first, priority classes, or fair shares across tenants.
Request preemption
Interrupting a running request so a waiting or higher-priority one can run, usually by evicting its cache to be swapped back or recomputed later.
Tail latency
The latency of the slowest requests, measured at p95 or p99. Batching choices that improve the average can make the tail worse.
Throughput-versus-latency trade-off
Bigger batches serve more tokens per second in total but slow every individual stream. Every deployment picks a point on this curve.