OpenAI, released 2025-08-04. Sparse MoE decoder, 21B total, 3.6B active, 128K context. Attention: GQA with alternating sliding-window and global layers.
Sparse MoEJetBrains 🇨🇿 · June 1, 2026
JetBrains Mellum2 Thinking (12B-A2.5B)
JetBrains' Mellum2 Thinking is a compact reasoning-tuned sparse MoE for code and agentic workflows.
Key detail
Uses 64 experts with 8 active per token, a 1024-token sliding window, and YaRN scaling on full-attention layers.Total params
12B
Active / token
2.5B
Context
128K
Fact sheet
- Scale
- 12B total, 2.5B active (20.8% active)
- Context window
- 131,072 tokens
- Decoder type
- Sparse MoE
- Attention
- GQA with 3:1 sliding-window/full attention
- Layer mix
- 21 sliding-window + 7 full GQA
- KV cache / token (bf16)
- 56 KiB · Low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
21%
2.5B of 12B weights run for each token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Google, released 2026-04-02. Sparse MoE decoder, 25B total, 3.8B active, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Cohere, released 2026-06-05. Sparse MoE decoder, 30B total, 3B active, 256K context. Attention: 8:1 GQA with 3:1 sliding-window/global attention.
Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.