# Multi-Head Latent Attention (MLA)

> Keys and values are compressed into a small latent vector that is cached instead of full K/V.

Category: Attention

Instead of sharing KV heads (as GQA does), MLA down-projects each token's keys and values into a compact latent and stores only that latent in the cache. At attention time it is projected back up.

Introduced with DeepSeek V2 and popularised by DeepSeek V3/R1, MLA reported quality on par with or better than MHA while caching far less per token. A decoupled RoPE path handles positional information.

Many large MoE models released since 2025 (Kimi, GLM-5, Mistral Large 3 and others) adopted MLA or variants of it.

**Trade-off:** Very small KV cache, at the price of extra projections and a more complex implementation.

## Models using MLA (28)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [GLM-5.2 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-2.md) — Z.ai, 2026-06-17
- [Kimi K2.7 Code (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-7-code.md) — Moonshot AI, 2026-06-12
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24
- [Ling 2.6 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-6-1t.md) — InclusionAI, 2026-04-23
- [Kimi K2.6 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-6.md) — Moonshot AI, 2026-04-20
- [GLM-5.1 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-1.md) — Z.ai, 2026-04-07
- [Mistral Small 4 (119B)](https://llm-atlas-7or.pages.dev/models/mistral-small-4.md) — Mistral AI, 2026-03-16
- [Sarvam (105B)](https://llm-atlas-7or.pages.dev/models/sarvam-105b.md) — Sarvam AI, 2026-03-03
- [Ling 2.5 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-5-1t.md) — InclusionAI, 2026-02-15
- [GLM-5 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-744b.md) — Z.ai, 2026-02-11
- [LongCat-Flash-Lite (68.5B-A3B)](https://llm-atlas-7or.pages.dev/models/longcat-flash-lite-68-5b-a3b.md) — Meituan, 2026-01-28
- [Kimi K2.5 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-5.md) — Moonshot AI, 2026-01-27
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [Mistral Large 3 (673B)](https://llm-atlas-7or.pages.dev/models/mistral-large-3.md) — Mistral AI, 2025-12-02
- [DeepSeek V3.2 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3-2.md) — DeepSeek, 2025-12-01
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30
- [Kimi K2 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2.md) — Moonshot AI, 2025-07-10
- [DeepSeek R1 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-r1.md) — DeepSeek, 2025-01-20
- [DeepSeek V3 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3.md) — DeepSeek, 2024-12-26

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/mla/
Web page: https://llm-atlas-7or.pages.dev/concepts/mla/