# QK-Norm (QK-Norm)

> RMSNorm applied to queries and keys before attention to keep logits well-behaved.

Category: Positions & normalization

Normalising queries and keys (usually before RoPE) prevents attention logits from growing too large, a common source of loss spikes in large-scale training.

OLMo 2 and Qwen3 helped make it standard; it is now a default ingredient in many recipes.

**Trade-off:** A stabiliser rather than an efficiency trick, and it costs almost nothing.

## Models using QK-Norm (40)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [MiniMax M3.1-Flash (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-1-flash-428b.md) — MiniMax, 2026-10-19
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md) — Alibaba Cloud, 2026-08-14
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [BTL-3 (27B)](https://llm-atlas-7or.pages.dev/models/btl-3-27b.md) — Bad Theory Labs, 2026-07-20
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [MiniMax M3 (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-428b.md) — MiniMax, 2026-06-13
- [Gemma 4 (12B)](https://llm-atlas-7or.pages.dev/models/gemma-4-12b.md) — Google, 2026-06-03
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28
- [Tencent Hy3-preview (295B-A21B)](https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b.md) — Tencent, 2026-04-23
- [Gemma 4 (31B)](https://llm-atlas-7or.pages.dev/models/gemma-4-31b.md) — Google, 2026-04-02
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md) — Google, 2026-04-02
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02
- [MiniMax M2.7 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b.md) — MiniMax, 2026-03-18
- [Sarvam (30B)](https://llm-atlas-7or.pages.dev/models/sarvam-30b.md) — Sarvam AI, 2026-03-03
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [OLMo 3 (32B)](https://llm-atlas-7or.pages.dev/models/olmo-3-32b.md) — Ai2, 2025-11-20
- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md) — Ai2, 2025-11-20
- [MiniMax M2 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-230b.md) — MiniMax, 2025-10-23
- [Gemma 3 (270M)](https://llm-atlas-7or.pages.dev/models/gemma-3-270m.md) — Google, 2025-08-14
- [GLM-4.5 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-355b.md) — Z.ai, 2025-07-28
- [Qwen3 (235B-A22B)](https://llm-atlas-7or.pages.dev/models/qwen3-235b-a22b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (32B)](https://llm-atlas-7or.pages.dev/models/qwen3-32b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (4B)](https://llm-atlas-7or.pages.dev/models/qwen3-4b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (8B)](https://llm-atlas-7or.pages.dev/models/qwen3-8b.md) — Alibaba Cloud, 2025-04-28
- [Gemma 3 (27B)](https://llm-atlas-7or.pages.dev/models/gemma-3-27b.md) — Google, 2025-03-11
- [OLMo 2 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-2-7b.md) — Ai2, 2024-11-25

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/qk-norm/
Web page: https://llm-atlas-7or.pages.dev/concepts/qk-norm/