Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
DenseAlibaba Cloud 🇨🇳 · April 28, 2025
Qwen3 (8B)
Dense Qwen3 baseline used here to show how little OLMo 3 changed the overall decoder recipe.
Key detail
Reference Qwen3 dense stack with QK-Norm and 8 KV heads.Parameters
8B
Context
128K
KV / token
144 KiB
Fact sheet
- Scale
- 8B parameters
- Context window
- 128,000 tokens
- Decoder type
- Dense
- Attention
- GQA with QK-Norm
- Layer mix
- 36 GQA
- KV cache / token (bf16)
- 144 KiB · Moderate
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source6
- Scientific
- 1
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Alibaba Cloud, released 2025-04-28. Dense decoder, 4B parameters, 32K context. Attention: GQA with QK-Norm.
NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.