Google, released 2026-04-02. Sparse MoE decoder, 25B total, 3.8B active, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Sparse MoEOpenAI 🇺🇸 · August 4, 2025
GPT-OSS (20B)
OpenAI's smaller open-weight MoE model favors width and alternating local/global attention.
Key detail
Wider and shallower than Qwen3, with attention bias and sink mechanisms.Total params
21B
Active / token
3.6B
Context
128K
Fact sheet
- Scale
- 21B total, 3.6B active (17.1% active)
- Context window
- 128,000 tokens
- Decoder type
- Sparse MoE
- Attention
- GQA with alternating sliding-window and global layers
- Layer mix
- 12 sliding-window + 12 global
- KV cache / token (bf16)
- 48 KiB · Low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
17%
3.6B of 21B weights run for each token.
AA Intelligence Index v4.3.2
Source9
- General
- 12.8
- Scientific
- 6.2
- Coding
- 19.4
- Agents
- 0
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Cohere, released 2026-06-05. Sparse MoE decoder, 30B total, 3B active, 256K context. Attention: 8:1 GQA with 3:1 sliding-window/global attention.
Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Poolside, released 2026-07-02. Sparse MoE decoder, 33B total, 3B active, 256K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.