# Sliding Window Attention (SWA)

> Some layers only attend to a local window of recent tokens instead of the whole context.

Category: Attention

Local (sliding-window) layers restrict each token to the last N tokens, so their KV cache stops growing once the window is full. A few global layers are kept so information can still travel across the whole context.

Gemma 3 made the pattern famous with a 5:1 local-to-global ratio and a 1,024-token window. The ratio and window size are the main knobs, and it is very often paired with GQA.

**Trade-off:** Cheap long contexts; the risk is weaker long-range recall if too few global layers remain.

## Models using SWA (38)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Naive-N0.5-Flash (309B)](https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b.md) — NaiveAI, 2026-09-27
- [Xiaomi MiMo-V2.6-Flash-RL (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b.md) — Xiaomi, 2026-09-21
- [Xiaomi MiMo-V2.6-Pro-RL (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t.md) — Xiaomi, 2026-09-21
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [DeepSeek V4.1-Flash (552B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash.md) — DeepSeek, 2026-09-10
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [North Mini Code (30B-A3B)](https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b.md) — Cohere, 2026-06-05
- [Gemma 4 (12B)](https://llm-atlas-7or.pages.dev/models/gemma-4-12b.md) — Google, 2026-06-03
- [JetBrains Mellum2 Thinking (12B-A2.5B)](https://llm-atlas-7or.pages.dev/models/jetbrains-mellum2-thinking-12b-a2-5b.md) — JetBrains, 2026-06-01
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28
- [Xiaomi MiMo-V2.5 (310B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-310b.md) — Xiaomi, 2026-04-22
- [Xiaomi MiMo-V2.5-Pro (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-pro-1-02t.md) — Xiaomi, 2026-04-22
- [Gemma 4 (31B)](https://llm-atlas-7or.pages.dev/models/gemma-4-31b.md) — Google, 2026-04-02
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md) — Google, 2026-04-02
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02
- [Tiny Aya (3.35B)](https://llm-atlas-7or.pages.dev/models/tiny-aya-3-35b.md) — Cohere, 2026-02-13
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [Step 3.5 Flash (196B)](https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b.md) — StepFun, 2026-02-01
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [Xiaomi MiMo-V2-Flash (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-flash-309b.md) — Xiaomi, 2025-12-16
- [OLMo 3 (32B)](https://llm-atlas-7or.pages.dev/models/olmo-3-32b.md) — Ai2, 2025-11-20
- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md) — Ai2, 2025-11-20
- [Gemma 3 (270M)](https://llm-atlas-7or.pages.dev/models/gemma-3-270m.md) — Google, 2025-08-14
- [GPT-OSS (120B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-120b.md) — OpenAI, 2025-08-04
- [GPT-OSS (20B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-20b.md) — OpenAI, 2025-08-04
- [Mistral Small 3.1 (24B)](https://llm-atlas-7or.pages.dev/models/mistral-small-3-1-24b.md) — Mistral AI, 2025-03-18
- [Gemma 3 (27B)](https://llm-atlas-7or.pages.dev/models/gemma-3-27b.md) — Google, 2025-03-11

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/swa/
Web page: https://llm-atlas-7or.pages.dev/concepts/swa/