Google, released 2026-04-02. Dense decoder, 31B parameters, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
DenseAi2 🇺🇸 · November 20, 2025
OLMo 3 (32B)
Scaled-up OLMo 3 keeps the same block design but moves to grouped-query attention.
Key detail
Keeps post-norm while scaling width and applying YaRN only on global layers.Parameters
32B
Context
64K
KV / token
256 KiB
Fact sheet
- Scale
- 32B parameters
- Context window
- 65,536 tokens
- Decoder type
- Dense
- Attention
- GQA with QK-Norm and 3:1 sliding-window/global attention
- Layer mix
- 48 sliding-window + 16 global
- KV cache / token (bf16)
- 256 KiB · High
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source6.5
- Scientific
- 3.2
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
Google, released 2025-03-11. Dense decoder, 27B parameters, 128K context. Attention: GQA with QK-Norm and 5:1 sliding-window/global attention.
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.