Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.
DenseBOSS Zhipin 🇨🇳 · February 10, 2026
Nanbeige 4.1 (3B)
Small on-device oriented model that stays close to Llama 3.2 while nudging the scaling choices.
Key detail
Llama-like stack without tying input embeddings to the output layer.Parameters
3B
Context
256K
KV / token
64 KiB
Fact sheet
- Scale
- 3B parameters
- Context window
- 262,144 tokens
- Decoder type
- Dense
- Attention
- GQA
- Layer mix
- 32 GQA
- KV cache / token (bf16)
- 64 KiB · Low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source8.4
- Scientific
- 5.4
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Meta, released 2024-09-25. Dense decoder, 3B parameters, 128K context. Attention: GQA.
Weibo, released 2026-06-15. Dense decoder, 3B parameters, 128K context. Attention: 8:1 GQA with RoPE.
Cohere, released 2026-02-13. Dense decoder, 3.4B parameters, 8K context. Attention: GQA with 3:1 sliding-window attention.