Bad Theory Labs, released 2026-07-20. Dense hybrid decoder, 27B parameters, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.
Dense hybridNVIDIA 🇺🇸 · March 16, 2026
Nemotron 3 Nano (4B)
Compact on-device hybrid that compresses Nemotron Nano 9B v2 into a mostly Mamba-2 stack with only four attention layers.
Key detail
Uses a 42-layer stack with 21 Mamba-2 blocks, 17 ReLU² FFNs, and just 4 GQA layers.Parameters
4B
Context
256K
KV / token
16 KiB
Fact sheet
- Scale
- 4B parameters
- Context window
- 262,144 tokens
- Decoder type
- Dense hybrid
- Attention
- GQA with only 4 attention layers
- Layer mix
- 4 GQA + 21 Mamba-2 + 17 FFN
- KV cache / token (bf16)
- 16 KiB · Very low
- License
- NVIDIA Nemotron Open Model License
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v5.0
Source1.8
- Knowledge Work
- 0.2
- Coding
- 0
- Science
- 0
- General
- 7.1
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Alibaba Cloud, released 2026-08-14. Dense hybrid decoder, 27B parameters, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.
Liquid AI, released 2026-10-27. Dense hybrid decoder, 1.7B parameters, 256K context. Attention: ShortConv plus gated GQA with QK-Norm and sliding-window/global attention.
NVIDIA, released 2025-12-04. Hybrid MoE decoder, 30B total, 3B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.