# Latent MoE (Latent MoE)

> Routed experts operate in a smaller latent space instead of the full hidden width.

Category: Feed-forward & experts

Tokens are projected down before they enter the routed experts and projected back up afterwards. Experts become cheaper, so a model can afford more of them or activate more per token.

Nemotron 3 Super and Ultra, Kimi K3 and Muse Spark Open use this layout with different compression ratios and normalization details.

**Trade-off:** Cheaper experts and communication; the bottleneck must be sized carefully.

## Models using Latent MoE (5)

- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md) — NVIDIA, 2026-10-15
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md) — NVIDIA, 2026-03-11

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/latent-moe/
Web page: https://llm-atlas-7or.pages.dev/concepts/latent-moe/