# 6. Distributed LLM inference

> When one GPU is not enough: splitting, copying and routing models across a cluster.

Track: Serving at scale. 29 terms.

Large models do not fit on one accelerator, and popular ones need far more throughput than one can provide. There are four basic ways to spread the work: copy the model (data parallelism), split each layer (tensor), split the stack of layers (pipeline), or spread the experts (expert parallelism).

Above the model sits a routing layer. Sending a request to a replica that already holds its prompt prefix in cache, keeping prefill and decode on separate pools, and refusing work before the system tips over all matter as much as raw GPU count.

**Interactive figure on the web page: Four ways to split a model.** A four-layer model on four GPUs. Each square is one quarter of a layer's weights. Switch modes to watch the pieces move: data parallelism copies everything, tensor parallelism slices every layer, pipeline parallelism stacks layers across GPUs, and expert parallelism spreads the experts.

## Terms

### Distributed inference

Running one model across several accelerators, or many copies across a cluster, because it does not fit on one device or one device is too slow.

### Model parallelism

Any scheme that splits a single model across devices: by tensor, by layer or by expert.

### Tensor parallelism

Splitting each weight matrix across GPUs so every layer runs on all of them at once, with an all-reduce after attention and the FFN. Low latency, but it needs very fast links such as NVLink, so it usually stays inside one server.

### Pipeline parallelism

Putting consecutive groups of layers on different devices and passing activations between them. Communication is light, but stages need a steady flow of micro-batches to avoid idle "bubbles".

### Data parallelism

Running full copies of the model on different devices, each handling different requests or training batches. It scales throughput, not the speed of any one request.

### Data-parallel serving

Data parallelism at inference time: several replicas behind a load balancer. Large MoE deployments often run attention data-parallel and experts expert-parallel.

### Expert parallelism

Placing different MoE experts on different GPUs, so each token is shipped with an all-to-all exchange to the GPUs that hold its chosen experts.

See: [Mixture of Experts](https://llm-atlas-7or.pages.dev/concepts/moe.md), [also in Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/learn/moe.md)

### Model sharding

Physically splitting the weights into pieces (shards) stored on different devices, whatever parallelism scheme decides the split.

### Model replication

Keeping several identical copies of a model to serve more traffic or to survive failures.

### Multi-GPU inference

One model instance spanning several GPUs in a single server, usually with tensor parallelism over NVLink.

### Multi-node inference

One model instance spanning several servers, needed for the very largest models. It depends on InfiniBand or RoCE networking and usually combines pipeline or expert parallelism across nodes.

### Distributed model serving

The whole system for serving models on a cluster: replicas, routers, autoscaling, caches and health checks.

### Distributed request routing

Choosing which replica or worker handles each incoming request.

### Cache-aware routing

Sending a request to the replica that already holds its prefix in KV cache, so it skips recomputing that prefill.

### Prefix-aware routing

Cache-aware routing that hashes or matches prompt prefixes, such as the system prompt and conversation so far, to choose a replica.

### Cluster scheduling

Allocating GPUs and nodes to models and jobs across a cluster, for example with Kubernetes and GPU-aware schedulers.

### Global routing

Routing across regions or clusters, weighing latency to the user, spare capacity and cost.

### Load balancing

Spreading requests across replicas to avoid hot spots. LLM balancers should look at queue depth and cache usage, not just request counts, because requests vary so much in size.

### Capacity-aware routing

Routing that uses live capacity signals such as free cache blocks, queue length or tokens in flight.

### Prefill/decode disaggregation

Running prefill and decode on separate GPU pools, because one is compute-bound and the other bandwidth-bound, and transferring the KV cache between them. It stops long prompts from stalling streams already in progress.

### Disaggregated serving

The general design of splitting inference stages (prefill, decode, sometimes attention and experts) onto hardware pools that scale independently.

### Distributed KV caching

Storing and sharing KV caches across machines, in CPU memory, SSDs or a dedicated cache tier, so a prefix computed once can be reused cluster-wide.

### High-speed interconnects

Links like NVLink and NVSwitch inside a server and InfiniBand or RoCE between servers. Their bandwidth decides which parallelism schemes are practical.

### Multi-tenant inference

Serving many customers or applications on shared infrastructure, with isolation, quotas and fairness between them.

### Inference quality of service (QoS)

Latency and throughput guarantees or priorities per tenant or request class, such as interactive traffic ahead of batch jobs.

### Admission control

Rejecting or delaying requests at the edge when the system is saturated, so the requests that are accepted still meet their latency targets.

### Autoscaling

Adding or removing replicas as load changes. Loading large weights takes minutes, so predictive scaling and warm pools matter.

### High availability

Staying up through failures with redundant replicas, health checks, retries and failover across zones.

### Accelerator-aware scheduling

Placing workloads according to accelerator type, memory and interconnect, for example putting a large model only on nodes with eight NVLinked GPUs.

Web page: https://llm-atlas-7or.pages.dev/learn/distributed/

_Definitions written for LLM Atlas._