Topic 6 · Serving at scale
Distributed LLM inference
When one GPU is not enough: splitting, copying and routing models across a cluster.
Large models do not fit on one accelerator, and popular ones need far more throughput than one can provide. There are four basic ways to spread the work: copy the model (data parallelism), split each layer (tensor), split the stack of layers (pipeline), or spread the experts (expert parallelism).
Above the model sits a routing layer. Sending a request to a replica that already holds its prompt prefix in cache, keeping prefill and decode on separate pools, and refusing work before the system tips over all matter as much as raw GPU count.
In this topic
29
terms, each defined below
Architecture concepts in the atlas
Four ways to split a model
A four-layer model on four GPUs. Each square is one quarter of a layer's weights. Switch modes to watch the pieces move: data parallelism copies everything, tensor parallelism slices every layer, pipeline parallelism stacks layers across GPUs, and expert parallelism spreads the experts.
GPU 0
GPU 1
GPU 2
GPU 3
Each layer's matrices are sliced into four. All GPUs work on every token at once and combine their partial results, which needs very fast links.
- Weights per GPU
- 25% per GPU
- Communication
- All-reduce after every attention and FFN
- Best for
- Lower latency inside one server (NVLink)
Glossary
29 terms
Distributed inference
Running one model across several accelerators, or many copies across a cluster, because it does not fit on one device or one device is too slow.
Model parallelism
Any scheme that splits a single model across devices: by tensor, by layer or by expert.
Tensor parallelism
Splitting each weight matrix across GPUs so every layer runs on all of them at once, with an all-reduce after attention and the FFN. Low latency, but it needs very fast links such as NVLink, so it usually stays inside one server.
Pipeline parallelism
Putting consecutive groups of layers on different devices and passing activations between them. Communication is light, but stages need a steady flow of micro-batches to avoid idle "bubbles".
Data parallelism
Running full copies of the model on different devices, each handling different requests or training batches. It scales throughput, not the speed of any one request.
Data-parallel serving
Data parallelism at inference time: several replicas behind a load balancer. Large MoE deployments often run attention data-parallel and experts expert-parallel.
Expert parallelism
Placing different MoE experts on different GPUs, so each token is shipped with an all-to-all exchange to the GPUs that hold its chosen experts.
Model sharding
Physically splitting the weights into pieces (shards) stored on different devices, whatever parallelism scheme decides the split.
Model replication
Keeping several identical copies of a model to serve more traffic or to survive failures.
Multi-GPU inference
One model instance spanning several GPUs in a single server, usually with tensor parallelism over NVLink.
Multi-node inference
One model instance spanning several servers, needed for the very largest models. It depends on InfiniBand or RoCE networking and usually combines pipeline or expert parallelism across nodes.
Distributed model serving
The whole system for serving models on a cluster: replicas, routers, autoscaling, caches and health checks.
Distributed request routing
Choosing which replica or worker handles each incoming request.
Cache-aware routing
Sending a request to the replica that already holds its prefix in KV cache, so it skips recomputing that prefill.
Prefix-aware routing
Cache-aware routing that hashes or matches prompt prefixes, such as the system prompt and conversation so far, to choose a replica.
Cluster scheduling
Allocating GPUs and nodes to models and jobs across a cluster, for example with Kubernetes and GPU-aware schedulers.
Global routing
Routing across regions or clusters, weighing latency to the user, spare capacity and cost.
Load balancing
Spreading requests across replicas to avoid hot spots. LLM balancers should look at queue depth and cache usage, not just request counts, because requests vary so much in size.
Capacity-aware routing
Routing that uses live capacity signals such as free cache blocks, queue length or tokens in flight.
Prefill/decode disaggregation
Running prefill and decode on separate GPU pools, because one is compute-bound and the other bandwidth-bound, and transferring the KV cache between them. It stops long prompts from stalling streams already in progress.
Disaggregated serving
The general design of splitting inference stages (prefill, decode, sometimes attention and experts) onto hardware pools that scale independently.
Distributed KV caching
Storing and sharing KV caches across machines, in CPU memory, SSDs or a dedicated cache tier, so a prefix computed once can be reused cluster-wide.
High-speed interconnects
Links like NVLink and NVSwitch inside a server and InfiniBand or RoCE between servers. Their bandwidth decides which parallelism schemes are practical.
Multi-tenant inference
Serving many customers or applications on shared infrastructure, with isolation, quotas and fairness between them.
Inference quality of service (QoS)
Latency and throughput guarantees or priorities per tenant or request class, such as interactive traffic ahead of batch jobs.
Admission control
Rejecting or delaying requests at the edge when the system is saturated, so the requests that are accepted still meet their latency targets.
Autoscaling
Adding or removing replicas as load changes. Loading large weights takes minutes, so predictive scaling and warm pools matter.
High availability
Staying up through failures with redundant replicas, health checks, retries and failover across zones.
Accelerator-aware scheduling
Placing workloads according to accelerator type, memory and interconnect, for example putting a large model only on nodes with eight NVLinked GPUs.