Architecture diagram of OLMo 2 (7B) — Figure 10: OLMo 2 7B · diagram by Sebastian Raschka · click to zoom

DenseAi2 🇺🇸 · November 25, 2024

OLMo 2 (7B)

Transparent dense model that keeps classic MHA and pushes normalization changes for training stability.

Key detail

Uses inside-residual post-norm instead of the usual pre-norm layout.

Parameters

7B

Context

4K

KV / token

512 KiB

Fact sheet

Scale: 7B parameters
Context window: 4,096 tokens
Decoder type: Dense
Attention: MHA with QK-Norm
Layer mix: 32 MHA
KV cache / token (bf16): 512 KiB · Very high
License: Apache License 2.0
Released: November 25, 2024

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

512 KiB

Smaller than 3% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

5.6

Architecture concepts

Sources & resources

View in article config.json Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

OLMo 3 (7B)
Ai2, released 2025-11-20. Dense decoder, 7B parameters, 64K context. Attention: MHA with QK-Norm and 3:1 sliding-window/global attention.
Qwen3 (8B)
Alibaba Cloud, released 2025-04-28. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm.
Gemma 4 (E4B)
Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Gemma 4 (E2B)
Google, released 2026-04-02. Dense decoder, 5.1B parameters, 128K context. Attention: Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention.