Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.
DenseWeibo 🇨🇳 · June 15, 2026
VibeThinker-3B (3B)
Qwen2.5-Coder-based dense reasoner whose gains mainly come from verifier-driven post-training rather than a new block design.
Key detail
Standard Qwen2.5-Coder-style dense stack strengthened by curriculum SFT, MGPO RL, Long2Short RL, offline self-distillation, and Instruct RL.Parameters
3B
Context
128K
KV / token
36 KiB
Fact sheet
- Scale
- 3B parameters
- Context window
- 131,072 tokens
- Decoder type
- Dense
- Attention
- 8:1 GQA with RoPE
- Layer mix
- 36 GQA
- KV cache / token (bf16)
- 36 KiB · Low
- License
- MIT License
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Meta, released 2024-09-25. Dense decoder, 3B parameters, 128K context. Attention: GQA.
BOSS Zhipin, released 2026-02-10. Dense decoder, 3B parameters, 256K context. Attention: GQA.
Cohere, released 2026-02-13. Dense decoder, 3.4B parameters, 8K context. Attention: GQA with 3:1 sliding-window attention.