NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.
DenseBOSS Zhipin 🇨🇳 · July 21, 2026
Nanbeige 4.2 (3B)
Compact agentic model that runs the same 22-layer dense transformer stack twice, resulting in 44 transformer blocks (layers) without duplicating the layer weights.
Key detail
Reuses the same 22-layer stack for a second pass, giving 44 block applications with 3B non-embedding parameters.Parameters
4B
Context
256K
KV / token
176 KiB
Fact sheet
- Scale
- 4B total, 3B non-embedding
- Context window
- 262,144 tokens
- Decoder type
- Dense
- Attention
- GQA with RoPE
- Layer mix
- 22 shared GQA layers × 2 passes
- KV cache / token (bf16)
- 176 KiB · High
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Alibaba Cloud, released 2025-04-28. Dense decoder, 4B parameters, 32K context. Attention: GQA with QK-Norm.
Cohere, released 2026-02-13. Dense decoder, 3.4B parameters, 8K context. Attention: GQA with 3:1 sliding-window attention.
Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.