Ornith, released 2026-08-23. Hybrid MoE decoder, 35B total, 3B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.
Hybrid MoENVIDIA 🇺🇸 · August 11, 2026
Nemotron 3.5 Lightning (30B-A3B)
NVIDIA's 30B hybrid model keeps 3B parameters active and extends the Nemotron 3 Nano backbone with a 1M-token context window and multi-token prediction.
Key detail
Adds a shared GQA-plus-MoE MTP predictor to the 52-layer Nemotron 3 Nano backbone.Total params
30B
Active / token
3B
Context
1M
Fact sheet
- Scale
- 30B total, 3B active (10% active)
- Context window
- 1,048,576 tokens
- Decoder type
- Hybrid MoE
- Attention
- Mostly Mamba-2 with six GQA layers
- Layer mix
- 6 GQA + 23 Mamba-2 + 23 MoE; GQA + MoE MTP predictor
- KV cache / token (bf16)
- 6 KiB · Very low
- License
- OpenMDW 1.1
- Released
Efficiency
Parameters active per token
10%
3B of 30B weights run for each token.
AA Intelligence Index v5.0
Source5.5
- Knowledge Work
- 1
- Coding
- 0.2
- Science
- 0.6
- General
- 20.6
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
NVIDIA, released 2026-03-11. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.
NVIDIA, released 2026-10-15. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with eight GQA layers.
Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).