Upstage, released 2026-07-22. Hybrid MoE decoder, 250B total, 15B active, 1M context. Attention: 3:1 Kimi Delta Attention and gated GQA with NoPE.
Hybrid MoENVIDIA 🇺🇸 · June 4, 2026
Nemotron 3 Ultra (550B-A55B)
NVIDIA's Ultra model scales the Nemotron 3 hybrid stack to 550B total parameters while keeping 55B active per token.
Key detail
Scales the latent-MoE hybrid to 108 layers with 48 Mamba-2, 48 latent MoE, and 12 GQA layers.Total params
550B
Active / token
55B
Context
1M
Fact sheet
- Scale
- 550B total, 55B active (10% active)
- Context window
- 1,000,000 tokens
- Decoder type
- Hybrid MoE
- Attention
- Mostly Mamba-2 with a few GQA layers
- Layer mix
- 12 GQA + 48 Mamba-2 + 48 MoE
- KV cache / token (bf16)
- 12 KiB · Very low
- License
- NVIDIA Nemotron Open Model License
- Released
Efficiency
Parameters active per token
10%
55B of 550B weights run for each token.
AA Intelligence Index v5.0
Source11.8
- Knowledge Work
- 12.9
- Coding
- 0.9
- Science
- 6
- General
- 27.4
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).
NVIDIA, released 2026-03-11. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.
NVIDIA, released 2026-10-15. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with eight GQA layers.