# OLMo 2 (7B)

> Transparent dense model that keeps classic MHA and pushes normalization changes for training stability.

| Property | Value |
| --- | --- |
| Lab | Ai2 (United States) |
| Released | 2024-11-25 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 7B parameters |
| Context window | 4,096 tokens |
| Attention | MHA with QK-Norm |
| Layer mix | 32 MHA |
| KV cache / token (bf16) | 512 KiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Uses inside-residual post-norm instead of the usual pre-norm layout.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.6

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/olmo-2-7b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A723-olmo-2-summary
- config.json: https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2501.00656
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-olmo-2-7b
- Web page: https://llm-atlas-7or.pages.dev/models/olmo-2-7b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/olmo-2-7b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._