# 7. Mixture of Experts (MoE)

> A huge model that only wakes up a small part of itself for each token.

Track: Serving at scale. 25 terms.

In a Mixture-of-Experts layer, the single feed-forward network is replaced by many smaller expert networks and a router. For every token the router picks a handful of experts, so a model can hold hundreds of billions of parameters while each token only pays for a few tens of billions.

The catch moves to systems. All experts must sit in memory, tokens have to be shuffled to wherever their experts live, and an unbalanced router leaves some GPUs overloaded and others idle. Most large open-weight models in the gallery are now MoEs.

**Interactive figure on the web page: A router at work.** Tokens arrive one by one. The router scores all eight experts and sends each token to its top two. Turn load balancing off to watch a few experts get swamped while others sit idle; the load bars underneath show the imbalance.

## Terms

### Mixture of Experts (MoE)

An architecture where the FFN in each block is replaced by many expert FFNs plus a router that sends each token to only a few of them. Total parameters grow without a matching rise in compute per token.

See: [Mixture of Experts](https://llm-atlas-7or.pages.dev/concepts/moe.md), [Latent MoE](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md), [related](https://llm-atlas-7or.pages.dev/models/?type=moe)

### Dense models

Models where every parameter is used for every token. Simpler to train and serve, but compute per token rises with total size.

See: [related](https://llm-atlas-7or.pages.dev/models/?type=dense)

### Sparse MoE models

MoE models where only a small subset of experts runs per token, for example 8 of 256. Most of the largest open-weight models are now sparse MoEs.

See: [related](https://llm-atlas-7or.pages.dev/models/?type=moe)

### Experts

The individual FFNs inside an MoE layer. Nobody assigns them topics; any specialisation emerges in training and tends to follow token types and syntax more than subjects.

See: [Mixture of Experts](https://llm-atlas-7or.pages.dev/concepts/moe.md)

### Expert MLPs

The MLP each expert is, typically a SwiGLU FFN much narrower than the dense FFN it replaces. Many designs also add one or more shared experts that every token uses.

See: [Latent MoE](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md)

### MoE routing

Deciding which experts process each token at each MoE layer.

### Router network

A small linear layer that maps each token's hidden state to one score per expert.

### Routing logits

The raw per-expert scores from the router, before softmax or sigmoid. Top-k is taken over them.

### Gating network

Another name for the router. "Gate" stresses that it also sets how much each chosen expert's output counts.

### Gating weights

The normalised router scores of the selected experts, used to weight each expert's output when combining them.

### Top-k expert selection

Choosing the k experts with the highest scores for each token. k is small, often between 1 and 8, however many experts the layer has.

### Top-2 routing

Top-k with k = 2, used by Mixtral and many early MoEs. Each token's output is the weighted sum of two experts.

### Active parameters

The parameters actually used for one token: attention, shared parts and the chosen experts. They drive compute per token and are the "A22B" in names like 235B-A22B.

See: [related](https://llm-atlas-7or.pages.dev/models/?sort=sparsest)

### Total parameters

All parameters, every expert included. They drive how much memory is needed to hold the model, even though most sit idle for any one token.

See: [related](https://llm-atlas-7or.pages.dev/models/?sort=largest)

### Expert dispatch

Sending tokens to their chosen experts: regrouping them by expert and, under expert parallelism, moving them to other GPUs.

### Expert combine

Gathering expert outputs back into the original token order and summing them with the gating weights.

### Token permutation

Reordering a batch so that tokens for the same expert sit next to each other, which allows efficient grouped matrix multiplications; it is reversed after the combine.

### Grouped matrix multiplication

One kernel that runs many independent matrix multiplications of different sizes, one per expert, instead of launching each separately. Essential for fast MoE layers.

### Expert load balancing

Keeping experts evenly used, through auxiliary balancing losses, per-expert router biases (as in DeepSeek's auxiliary-loss-free method) or capacity limits.

### Expert placement

Deciding which experts live on which GPUs. Popular experts can be replicated to spread their load.

### Inter-GPU communication

Data exchanged between GPUs during a forward pass: all-reduce for tensor parallelism, all-to-all for expert parallelism. It often caps MoE throughput.

### All-to-all communication

A collective operation where every GPU sends a different piece of data to every other GPU. Used to dispatch tokens to experts and bring the results back.

### Routing imbalance

When the router sends far more tokens to some experts than others, creating stragglers, dropped tokens or idle GPUs.

### Sparse computation

Doing only a fraction of the possible work for each input. MoE does it for the FFN; sparse attention applies the same idea to attention.

See: [DeepSeek Sparse Attention](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md), [Qwen Sparse Attention](https://llm-atlas-7or.pages.dev/concepts/qwen-sparse-attention.md)

### Expert parallelism

Spreading experts across GPUs: each holds a subset and receives the tokens routed to it through all-to-all. Large deployments pair it with data-parallel attention.

See: [related](https://llm-atlas-7or.pages.dev/learn/distributed/#expert-parallelism), [also in Distributed LLM inference](https://llm-atlas-7or.pages.dev/learn/distributed.md)

Web page: https://llm-atlas-7or.pages.dev/learn/moe/

_Definitions written for LLM Atlas._