# Multi-Head Attention (MHA)

> The original transformer attention: every head has its own queries, keys and values.

Category: Attention

Self-attention lets every token build its representation from a weighted mix of all earlier tokens. The weights come from comparing a token's query with every key; the values are what gets mixed.

Multi-head attention runs several of these attention operations side by side, each in a smaller subspace, so different heads can track different relationships (syntax, coreference, position) at once. Their outputs are concatenated and projected back.

In a decoder, a causal mask hides future tokens. Because every head stores its own keys and values, MHA has the largest KV cache of the common variants.

**Trade-off:** Maximum expressiveness per layer, but the KV cache grows with heads × layers × tokens.

## Models using MHA (4)

- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md) — Ai2, 2025-11-20
- [Ouro-Thinking (2.6B)](https://llm-atlas-7or.pages.dev/models/ouro-thinking-2-6b.md) — ByteDance, 2025-10-29
- [OLMo 2 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-2-7b.md) — Ai2, 2024-11-25
- [GPT-2 XL (1.5B)](https://llm-atlas-7or.pages.dev/models/gpt-2-xl-1-5b.md) — OpenAI, 2019-11-05

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/mha/
Web page: https://llm-atlas-7or.pages.dev/concepts/mha/