Ai2, released 2024-11-25. Dense decoder, 7B parameters, 4K context. Attention: MHA with QK-Norm.
DenseAi2 🇺🇸 · November 20, 2025
OLMo 3 (7B)
New transparent Allen AI model that keeps OLMo's post-norm flavor while modernizing context handling.
Key detail
Retains post-norm, keeps MHA, and applies YaRN only on global layers.Parameters
7B
Context
64K
KV / token
512 KiB
Fact sheet
- Scale
- 7B parameters
- Context window
- 65,536 tokens
- Decoder type
- Dense
- Attention
- MHA with QK-Norm and 3:1 sliding-window/global attention
- Layer mix
- 24 sliding-window + 8 global
- KV cache / token (bf16)
- 512 KiB · Very high
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source5.2
- Scientific
- 2.9
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Google, released 2026-04-02. Dense decoder, 5.1B parameters, 128K context. Attention: Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention.
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.