Topic 7 · Serving at scale
Mixture of Experts (MoE)
A huge model that only wakes up a small part of itself for each token.
In a Mixture-of-Experts layer, the single feed-forward network is replaced by many smaller expert networks and a router. For every token the router picks a handful of experts, so a model can hold hundreds of billions of parameters while each token only pays for a few tens of billions.
The catch moves to systems. All experts must sit in memory, tokens have to be shuffled to wherever their experts live, and an unbalanced router leaves some GPUs overloaded and others idle. Most large open-weight models in the gallery are now MoEs.
In this topic
25
terms, each defined below
Architecture concepts in the atlas
A router at work
Tokens arrive one by one. The router scores all eight experts and sends each token to its top two. Turn load balancing off to watch a few experts get swamped while others sit idle; the load bars underneath show the imbalance.
tokens routed to each expert
- Expert FFN weights used per token
- 2 of 8 = 25%
- Tokens routed
- 0
- Busiest expert vs average
- 1.00×
Glossary
25 terms
Mixture of Experts (MoE)
An architecture where the FFN in each block is replaced by many expert FFNs plus a router that sends each token to only a few of them. Total parameters grow without a matching rise in compute per token.
Dense models
Models where every parameter is used for every token. Simpler to train and serve, but compute per token rises with total size.
Sparse MoE models
MoE models where only a small subset of experts runs per token, for example 8 of 256. Most of the largest open-weight models are now sparse MoEs.
Experts
The individual FFNs inside an MoE layer. Nobody assigns them topics; any specialisation emerges in training and tends to follow token types and syntax more than subjects.
Expert MLPs
The MLP each expert is, typically a SwiGLU FFN much narrower than the dense FFN it replaces. Many designs also add one or more shared experts that every token uses.
MoE routing
Deciding which experts process each token at each MoE layer.
Router network
A small linear layer that maps each token's hidden state to one score per expert.
Routing logits
The raw per-expert scores from the router, before softmax or sigmoid. Top-k is taken over them.
Gating network
Another name for the router. "Gate" stresses that it also sets how much each chosen expert's output counts.
Gating weights
The normalised router scores of the selected experts, used to weight each expert's output when combining them.
Top-k expert selection
Choosing the k experts with the highest scores for each token. k is small, often between 1 and 8, however many experts the layer has.
Top-2 routing
Top-k with k = 2, used by Mixtral and many early MoEs. Each token's output is the weighted sum of two experts.
Active parameters
The parameters actually used for one token: attention, shared parts and the chosen experts. They drive compute per token and are the "A22B" in names like 235B-A22B.
Total parameters
All parameters, every expert included. They drive how much memory is needed to hold the model, even though most sit idle for any one token.
Expert dispatch
Sending tokens to their chosen experts: regrouping them by expert and, under expert parallelism, moving them to other GPUs.
Expert combine
Gathering expert outputs back into the original token order and summing them with the gating weights.
Token permutation
Reordering a batch so that tokens for the same expert sit next to each other, which allows efficient grouped matrix multiplications; it is reversed after the combine.
Grouped matrix multiplication
One kernel that runs many independent matrix multiplications of different sizes, one per expert, instead of launching each separately. Essential for fast MoE layers.
Expert load balancing
Keeping experts evenly used, through auxiliary balancing losses, per-expert router biases (as in DeepSeek's auxiliary-loss-free method) or capacity limits.
Expert placement
Deciding which experts live on which GPUs. Popular experts can be replicated to spread their load.
Inter-GPU communication
Data exchanged between GPUs during a forward pass: all-reduce for tensor parallelism, all-to-all for expert parallelism. It often caps MoE throughput.
All-to-all communication
A collective operation where every GPU sends a different piece of data to every other GPU. Used to dispatch tokens to experts and bring the results back.
Routing imbalance
When the router sends far more tokens to some experts than others, creating stragglers, dropped tokens or idle GPUs.
Sparse computation
Doing only a fraction of the possible work for each input. MoE does it for the FFN; sparse attention applies the same idea to attention.
Expert parallelism
Spreading experts across GPUs: each holds a subset and receives the tokens routed to it through all-to-all. Large deployments pair it with data-parallel attention.