# LLM Atlas

> Browse, filter and compare 100+ open-weight LLM architectures: diagrams, attention mechanisms, MoE layouts, KV-cache sizes and fact sheets, sourced from Sebastian Raschka's LLM Architecture Gallery.

Source of truth: Sebastian Raschka's LLM Architecture Gallery — https://sebastianraschka.com/llm-architecture-gallery/. All architecture diagrams and model facts originate there; this site re-presents them. Data synced 2026-11-04. 118 models from 44 labs, released 2019-11-05 to 2026-10-29.

Every page is static HTML. Machine-readable alternatives: a Markdown file per model and concept (append `.md` to the page path without the trailing slash) and a JSON API (no auth, CORS-safe static files).

## API

- [API index](https://llm-atlas-7or.pages.dev/api/index.json): list of all JSON endpoints
- [All models (JSON)](https://llm-atlas-7or.pages.dev/api/models.json): array with every field shown on the site
- [All concepts (JSON)](https://llm-atlas-7or.pages.dev/api/concepts.json)
- [Full text (Markdown)](https://llm-atlas-7or.pages.dev/llms-full.txt): every model and concept fact sheet in one file
- [RSS feed](https://llm-atlas-7or.pages.dev/feed.xml): newest architectures

## Pages

- [Models](https://llm-atlas-7or.pages.dev/models/): searchable gallery; supports ?q=, ?type=dense|moe|hybrid-moe|hybrid|recurrent, ?concept=<slug>, ?lab=<name>, ?sort=newest|oldest|largest|smallest|sparsest|kv|context|name
- [Compare](https://llm-atlas-7or.pages.dev/compare/): ?a=<slug>&b=<slug> side-by-side diff (client-rendered; use the JSON API instead)
- [Concepts](https://llm-atlas-7or.pages.dev/concepts/): explainers for attention, MoE, normalization and residual-path ideas
- [Timeline](https://llm-atlas-7or.pages.dev/timeline/): models in release order
- [Insights](https://llm-atlas-7or.pages.dev/insights/): aggregate charts with data tables
- [Agent guide](https://llm-atlas-7or.pages.dev/agents/)

## Models

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md): Mistral AI, 2026-10-29, Sparse MoE, 1T total, 52B active (5.2% active), MLA with modified DeepSeek Sparse Attention, index reuse, and KV sharing
- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md): Liquid AI, 2026-10-27, Dense hybrid, 1.73B parameters, ShortConv plus gated GQA with QK-Norm and sliding-window/global attention
- [MiniMax M3.1-Flash (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-1-flash-428b.md): MiniMax, 2026-10-19, Sparse MoE, 428B total, 23B active (5.4% active), GQA with QK-Norm and MiniMax Sparse Attention
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md): Recursive, 2026-10-18, Sparse MoE, 39.8B total, 2.5B active (6.3% active), GQA with QK-Norm, learned attention sinks, and sliding-window/global attention
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md): Reflection, 2026-10-16, Sparse MoE, 501.3B total, 23.4B active (4.7% active), Gated GQA with QK-Norm and sliding-window/global attention
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md): NVIDIA, 2026-10-16, Dense, 4B parameters, Gated GQA; 3:1 sliding-window/full, RoPE
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md): Meta, 2026-10-15, Sparse MoE, 816B total, 42B active (5.1% active), Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention
- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md): NVIDIA, 2026-10-15, Hybrid MoE, 120B total, 12B active (10% active), Mostly Mamba-2 with eight GQA layers
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md): InclusionAI, 2026-10-13, Sparse hybrid, 560B total, 25B active (4.5% active), 7:1 Kimi Delta Attention and gated MLA
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md): Aleph Alpha, 2026-10-03, Sparse MoE, 78B total, 3.46B active (4.4% active), GQA with QK-Norm and 4:1 sliding-window/global attention
- [Naive-N0.5-Flash (309B)](https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b.md): NaiveAI, 2026-09-27, Sparse MoE, 309B total, 15.5B active (5.0% active), GQA with sliding-window attention and DeepSeek Sparse Attention
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md): Fireworks AI, 2026-09-23, Sparse hybrid, 2.8T total, 104B active (3.7% active), 3:1 Kimi Delta Attention and Gated MLA with NoPE
- [Xiaomi MiMo-V2.6-Flash-RL (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b.md): Xiaomi, 2026-09-21, Sparse omnimodal MoE, 309B total, 15B active (4.9% active), GQA with sliding-window/global attention
- [Xiaomi MiMo-V2.6-Pro-RL (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t.md): Xiaomi, 2026-09-21, Sparse omnimodal MoE, 1.02T total, 42B active (4.1% active), GQA with 6:1 sliding-window/global attention
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md): StepFun, 2026-09-20, Sparse MoE, 600B total, 27B active (4.5% active), Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention
- [DeepSeek V4.1-Flash (552B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash.md): DeepSeek, 2026-09-10, Sparse MoE (causal encoder-decoder), 552B backbone + 196B Engram; 8B active prefill, 16B active decode (2.9% of backbone), MLA-style CSA2 + SWA with cross-layer KV and sparse-index reuse
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md): Tencent, 2026-08-28, Sparse MoE, 770B backbone, 49B active (6.4% active); plus 10B MTP, Gated MLA with DeepSeek Sparse Attention and IndexCache
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md): Z.ai, 2026-08-26, Sparse hybrid, 320B total, 18B active (5.6% active), 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md): Alibaba Cloud, 2026-08-26, Sparse hybrid, 125B core, 6B active (4.8% active); plus 51B n-gram embeddings and 4B MTP, 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA)
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md): Ornith, 2026-08-23, Sparse hybrid, 35B total, 3B active (8.6% active), 3:1 Gated DeltaNet and Gated Attention
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md): Alibaba Cloud, 2026-08-14, Dense hybrid, 27B parameters, 3:1 Gated DeltaNet and Gated Attention
- [Nemotron 3.5 Lightning (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-lightning-30b-a3b.md): NVIDIA, 2026-08-11, Hybrid MoE, 30B total, 3B active (10% active), Mostly Mamba-2 with six GQA layers
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md): Meta, 2026-08-10, Dense, 29.6B parameters (including perception encoder), Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md): InclusionAI, 2026-08-04, Sparse hybrid, 124B total, 5.1B active (4.1% active), 5:1 Kimi Delta Attention and gated MLA
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md): Moonshot AI, 2026-07-27, Sparse hybrid, 2.8T total, 104B active (3.7% active), 3:1 Kimi Delta Attention and Gated MLA with NoPE
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md): Upstage, 2026-07-22, Sparse hybrid, 250B total, 15B active (6% active), 3:1 Kimi Delta Attention and gated GQA with NoPE
- [Antares (1B)](https://llm-atlas-7or.pages.dev/models/antares-1b.md): Cisco, 2026-07-21, Dense, 1B parameters, GQA with RoPE
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md): Poolside, 2026-07-21, Sparse MoE, 118B total, 8B active (6.8% active), Gated GQA with QK-Norm and 3:1 sliding-window/global attention
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md): Motif Technologies, 2026-07-21, Sparse MoE, 314B total, 13B active (4.1% active), GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention
- [Nanbeige 4.2 (3B)](https://llm-atlas-7or.pages.dev/models/nanbeige-4-2-3b.md): BOSS Zhipin, 2026-07-21, Dense, 4B total, 3B non-embedding, GQA with RoPE
- [BTL-3 (27B)](https://llm-atlas-7or.pages.dev/models/btl-3-27b.md): Bad Theory Labs, 2026-07-20, Dense hybrid, 27B base + 934 MB LoRA adapter, 3:1 Gated DeltaNet and Gated Attention
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md): Thinking Machines Lab, 2026-07-15, Sparse MoE, 975B total, 41B active (4.2% active), GQA with QK-Norm and learned relative-position bias in a 5:1 sliding-window/global pattern
- [Soofi S (30B-A3B)](https://llm-atlas-7or.pages.dev/models/soofi-s-30b-a3b.md): SOOFI Consortium, 2026-07-10, Hybrid MoE, 31.6B total, 3.2B active (10.1% active), Mostly Mamba-2 with 6 GQA layers and no positional embeddings (NoPE)
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md): Poolside, 2026-07-02, Sparse MoE, 33B total, 3B active (9.1% active), Gated GQA with QK-Norm and 3:1 sliding-window/global attention
- [GLM-5.2 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-2.md): Z.ai, 2026-06-17, Sparse MoE, 744B total, 40B active (5.4% active), MLA with DeepSeek Sparse Attention and IndexShare
- [VibeThinker-3B (3B)](https://llm-atlas-7or.pages.dev/models/vibethinker-3b.md): Weibo, 2026-06-15, Dense, 3B parameters, 8:1 GQA with RoPE
- [MiniMax M3 (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-428b.md): MiniMax, 2026-06-13, Sparse MoE, 428B total, 23B active (5.4% active), GQA with QK-Norm and MiniMax Sparse Attention
- [Kimi K2.7 Code (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-7-code.md): Moonshot AI, 2026-06-12, Sparse MoE, 1T total, 32B active (3.2% active), MLA
- [North Mini Code (30B-A3B)](https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b.md): Cohere, 2026-06-05, Sparse MoE, 30B total, 3B active (10% active), 8:1 GQA with 3:1 sliding-window/global attention
- [Nemotron 3 Ultra (550B-A55B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-ultra-550b-a55b.md): NVIDIA, 2026-06-04, Hybrid MoE, 550B total, 55B active (10% active), Mostly Mamba-2 with a few GQA layers
- [Gemma 4 (12B)](https://llm-atlas-7or.pages.dev/models/gemma-4-12b.md): Google, 2026-06-03, Dense, 12B parameters, GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- [JetBrains Mellum2 Thinking (12B-A2.5B)](https://llm-atlas-7or.pages.dev/models/jetbrains-mellum2-thinking-12b-a2-5b.md): JetBrains, 2026-06-01, Sparse MoE, 12B total, 2.5B active (20.8% active), GQA with 3:1 sliding-window/full attention
- [LFM2.5 (1.2B)](https://llm-atlas-7or.pages.dev/models/lfm2-5-1-2b.md): Liquid AI, 2026-05-28, Dense hybrid, 1.2B parameters, LIV convolution blocks plus GQA
- [LFM2.5 (350M)](https://llm-atlas-7or.pages.dev/models/lfm2-5-350m.md): Liquid AI, 2026-05-28, Dense hybrid, 350M parameters, LIV convolution blocks plus GQA
- [LFM2.5 (8B-A1B)](https://llm-atlas-7or.pages.dev/models/lfm2-5-8b-a1b.md): Liquid AI, 2026-05-28, Sparse hybrid, 8.3B total, 1.5B active (18.1% active), LIV convolution blocks plus GQA and MoE
- [Command A+ (218B-A25B)](https://llm-atlas-7or.pages.dev/models/command-a-218b-a25b.md): Cohere, 2026-05-20, Sparse MoE, 218B total, 25B active/token (11.5% active), 16:1 GQA with 3:1 sliding-window/global attention
- [ZAYA1-8B (8.4B)](https://llm-atlas-7or.pages.dev/models/zaya1-8b.md): Zyphra, 2026-05-06, Sparse MoE, 8.4B total, 760M active/token (9% active), CCA with 4:1 GQA, RoPE, and Q/K L2 norm
- [Granite 4.1 (30B)](https://llm-atlas-7or.pages.dev/models/granite-4-1-30b.md): IBM, 2026-04-29, Dense, 30B parameters, GQA with RoPE
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md): Poolside, 2026-04-28, Sparse MoE, 33B total, 3B active (9.1% active), Gated GQA with QK-Norm and 3:1 sliding-window/global attention
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md): DeepSeek, 2026-04-24, Sparse MoE, 284B total, 13B active (4.6% active), MLA-style CSA/HCA with mHC
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md): DeepSeek, 2026-04-24, Sparse MoE, 1.6T total, 49B active (3.1% active), MLA-style CSA/HCA with mHC
- [Ling 2.6 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-6-1t.md): InclusionAI, 2026-04-23, Sparse hybrid, 1T total, 63B active (6.3% active), Lightning Attention plus MLA
- [Tencent Hy3-preview (295B-A21B)](https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b.md): Tencent, 2026-04-23, Sparse MoE, 295B total, 21B active (7.1% active), GQA with QK-Norm
- [Qwen3.6 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-27b.md): Alibaba Cloud, 2026-04-22, Dense hybrid, 27B parameters, 3:1 Gated DeltaNet and Gated Attention
- [Xiaomi MiMo-V2.5 (310B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-310b.md): Xiaomi, 2026-04-22, Sparse omnimodal MoE, 310B total, 15B active (4.8% active), 5:1 sliding-window/global attention
- [Xiaomi MiMo-V2.5-Pro (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-pro-1-02t.md): Xiaomi, 2026-04-22, Sparse MoE, 1.02T total, 42B active (4.1% active), GQA with 6:1 sliding-window/global attention
- [Kimi K2.6 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-6.md): Moonshot AI, 2026-04-20, Sparse MoE, 1T total, 32B active (3.2% active), MLA
- [Qwen3.6 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-35b-a3b.md): Alibaba Cloud, 2026-04-15, Sparse hybrid, 35B total, 3B active (8.6% active), 3:1 Gated DeltaNet and Gated Attention
- [GLM-5.1 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-1.md): Z.ai, 2026-04-07, Sparse MoE, 744B total, 40B active (5.4% active), MLA with DeepSeek Sparse Attention
- [Gemma 4 (31B)](https://llm-atlas-7or.pages.dev/models/gemma-4-31b.md): Google, 2026-04-02, Dense, 30.7B parameters, GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md): Google, 2026-04-02, Sparse MoE, 25.2B total, 3.8B active (15.1% active), GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md): Google, 2026-04-02, Dense, 5.1B parameters (2.3B effective), Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md): Google, 2026-04-02, Dense, 8B parameters (4.5B effective), GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- [MiniMax M2.7 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b.md): MiniMax, 2026-03-18, Sparse MoE, 230B total, 10B active (4.3% active), GQA with QK-Norm
- [Mistral Small 4 (119B)](https://llm-atlas-7or.pages.dev/models/mistral-small-4.md): Mistral AI, 2026-03-16, Sparse MoE, 119B total, 6.63B active (5.6% active), MLA
- [Nemotron 3 Nano (4B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-4b.md): NVIDIA, 2026-03-16, Dense hybrid, 4B parameters, GQA with only 4 attention layers
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md): NVIDIA, 2026-03-11, Hybrid MoE, 120B total, 12B active (10% active), Mostly Mamba-2 with a few GQA layers
- [Sarvam (30B)](https://llm-atlas-7or.pages.dev/models/sarvam-30b.md): Sarvam AI, 2026-03-03, Sparse MoE, 30B total, 2.4B active (8% active), GQA with QK-Norm
- [Sarvam (105B)](https://llm-atlas-7or.pages.dev/models/sarvam-105b.md): Sarvam AI, 2026-03-03, Sparse MoE, 105B total, 10.3B active (9.8% active), MLA with KV LayerNorm and NoPE + RoPE
- [Qwen3.5 (397B)](https://llm-atlas-7or.pages.dev/models/qwen3-5-397b.md): Alibaba Cloud, 2026-02-16, Sparse hybrid, 397B total, 17B active (4.3% active), 3:1 Gated DeltaNet and Gated Attention
- [Ling 2.5 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-5-1t.md): InclusionAI, 2026-02-15, Sparse hybrid, 1T total, 63B active (6.3% active), Lightning Attention plus MLA
- [Tiny Aya (3.35B)](https://llm-atlas-7or.pages.dev/models/tiny-aya-3-35b.md): Cohere, 2026-02-13, Dense, 3.35B parameters, GQA with 3:1 sliding-window attention
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md): MiniMax, 2026-02-12, Sparse MoE, 230B total, 10B active (4.3% active), GQA with QK-Norm
- [GLM-5 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-744b.md): Z.ai, 2026-02-11, Sparse MoE, 744B total, 40B active (5.4% active), MLA with DeepSeek Sparse Attention
- [Nanbeige 4.1 (3B)](https://llm-atlas-7or.pages.dev/models/nanbeige-4-1-3b.md): BOSS Zhipin, 2026-02-10, Dense, 3B parameters, GQA
- [Step 3.5 Flash (196B)](https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b.md): StepFun, 2026-02-01, Sparse MoE, 196B total, 11B active (5.6% active), GQA with 3:1 sliding-window attention
- [LongCat-Flash-Lite (68.5B-A3B)](https://llm-atlas-7or.pages.dev/models/longcat-flash-lite-68-5b-a3b.md): Meituan, 2026-01-28, Sparse MoE, 68.5B total, ~3B active avg (4.4% active), MLA with RoPE + NoPE
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md): Arcee AI, 2026-01-27, Sparse MoE, 400B total, 13B active (3.3% active), GQA with gated attention and 3:1 sliding-window/global attention
- [Kimi K2.5 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-5.md): Moonshot AI, 2026-01-27, Sparse MoE, 1T total, 32B active (3.2% active), MLA
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md): Z.ai, 2025-12-22, Sparse MoE, 355B total, 32B active (9% active), GQA with QK-Norm
- [Xiaomi MiMo-V2-Flash (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-flash-309b.md): Xiaomi, 2025-12-16, Sparse MoE, 309B total, 15B active (4.9% active), 5:1 sliding-window/global attention
- [Nemotron 3 Nano (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-30b-a3b.md): NVIDIA, 2025-12-04, Hybrid MoE, 30B total, 3B active (10% active), Mostly Mamba-2 with a few GQA layers
- [Mistral Large 3 (673B)](https://llm-atlas-7or.pages.dev/models/mistral-large-3.md): Mistral AI, 2025-12-02, Sparse MoE, 673B total, 41B active (6.1% active), MLA
- [DeepSeek V3.2 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3-2.md): DeepSeek, 2025-12-01, Sparse MoE, 671B total, 37B active (5.5% active), MLA with DeepSeek Sparse Attention
- [INTELLECT-3 (106B)](https://llm-atlas-7or.pages.dev/models/intellect-3.md): Prime Intellect, 2025-11-26, Sparse MoE, 106B total, 12B active (11.3% active), GQA
- [OLMo 3 (32B)](https://llm-atlas-7or.pages.dev/models/olmo-3-32b.md): Ai2, 2025-11-20, Dense, 32B parameters, GQA with QK-Norm and 3:1 sliding-window/global attention
- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md): Ai2, 2025-11-20, Dense, 7B parameters, MHA with QK-Norm and 3:1 sliding-window/global attention
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md): Moonshot AI, 2025-10-30, Sparse hybrid, 48B total, 3B active (6.3% active), 3:1 Kimi Delta Attention and MLA
- [Ouro-Thinking (2.6B)](https://llm-atlas-7or.pages.dev/models/ouro-thinking-2-6b.md): ByteDance, 2025-10-29, Dense, 2.6B parameters, MHA with RoPE
- [MiniMax M2 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-230b.md): MiniMax, 2025-10-23, Sparse MoE, 230B total, 10B active (4.3% active), GQA with QK-Norm and partial RoPE
- [Qwen3 Next (80B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-next-80b-a3b.md): Alibaba Cloud, 2025-09-09, Sparse hybrid, 80B total, 3B active (3.8% active), 3:1 Gated DeltaNet and Gated Attention
- [Grok 2.5 (270B)](https://llm-atlas-7or.pages.dev/models/grok-2-5-270b.md): xAI, 2025-08-22, Sparse MoE, 270B parameters, GQA
- [Gemma 3 (270M)](https://llm-atlas-7or.pages.dev/models/gemma-3-270m.md): Google, 2025-08-14, Dense, 270M parameters, Multi-query attention with QK-Norm and 5:1 sliding-window/global attention
- [GPT-OSS (120B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-120b.md): OpenAI, 2025-08-04, Sparse MoE, 117B total, 5.1B active (4.4% active), GQA with alternating sliding-window and global layers
- [GPT-OSS (20B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-20b.md): OpenAI, 2025-08-04, Sparse MoE, 21B total, 3.6B active (17.1% active), GQA with alternating sliding-window and global layers
- [Qwen3 Coder Flash (30B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-coder-flash-30b-a3b.md): Alibaba Cloud, 2025-07-31, Sparse MoE, 30B total, 3.3B active (11% active), GQA
- [GLM-4.5 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-355b.md): Z.ai, 2025-07-28, Sparse MoE, 355B total, 32B active (9% active), GQA with QK-Norm
- [GLM-4.5-Air (106B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-air.md): Z.ai, 2025-07-28, Sparse MoE, 106B total, 12B active (11.3% active), GQA
- [Kimi K2 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2.md): Moonshot AI, 2025-07-10, Sparse MoE, 1T total, 32B active (3.2% active), MLA
- [SmolLM3 (3B)](https://llm-atlas-7or.pages.dev/models/smollm3-3b.md): Hugging Face, 2025-06-19, Dense, 3B parameters, GQA with periodic NoPE layers
- [Qwen3 (0.6B)](https://llm-atlas-7or.pages.dev/models/qwen3-0-6b.md): Alibaba Cloud, 2025-04-28, Dense, 0.6B parameters, GQA
- [Qwen3 (235B-A22B)](https://llm-atlas-7or.pages.dev/models/qwen3-235b-a22b.md): Alibaba Cloud, 2025-04-28, Sparse MoE, 235B total, 22B active (9.4% active), GQA with QK-Norm
- [Qwen3 (30B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-30b-a3b.md): Alibaba Cloud, 2025-04-28, Sparse MoE, 30B total, 3B active (10% active), GQA
- [Qwen3 (32B)](https://llm-atlas-7or.pages.dev/models/qwen3-32b.md): Alibaba Cloud, 2025-04-28, Dense, 32B parameters, GQA with QK-Norm
- [Qwen3 (4B)](https://llm-atlas-7or.pages.dev/models/qwen3-4b.md): Alibaba Cloud, 2025-04-28, Dense, 4B parameters, GQA with QK-Norm
- [Qwen3 (8B)](https://llm-atlas-7or.pages.dev/models/qwen3-8b.md): Alibaba Cloud, 2025-04-28, Dense, 8B parameters, GQA with QK-Norm
- [Llama 4 Maverick (400B)](https://llm-atlas-7or.pages.dev/models/llama-4-maverick.md): Meta, 2025-04-05, Sparse MoE, 400B total, 17B active (4.3% active), GQA
- [Mistral Small 3.1 (24B)](https://llm-atlas-7or.pages.dev/models/mistral-small-3-1-24b.md): Mistral AI, 2025-03-18, Dense, 24B parameters, Standard GQA
- [xLSTM (7B)](https://llm-atlas-7or.pages.dev/models/xlstm-7b.md): NXAI, 2025-03-17, Recurrent, 7B parameters, No self-attention; mLSTM recurrent layers with matrix memory
- [Gemma 3 (27B)](https://llm-atlas-7or.pages.dev/models/gemma-3-27b.md): Google, 2025-03-11, Dense, 27B parameters, GQA with QK-Norm and 5:1 sliding-window/global attention
- [DeepSeek R1 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-r1.md): DeepSeek, 2025-01-20, Sparse MoE, 671B total, 37B active (5.5% active), MLA
- [DeepSeek V3 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3.md): DeepSeek, 2024-12-26, Sparse MoE, 671B total, 37B active (5.5% active), MLA
- [Phi-4 (14B)](https://llm-atlas-7or.pages.dev/models/phi-4.md): Microsoft, 2024-12-12, Dense, 14B parameters, GQA with RoPE
- [OLMo 2 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-2-7b.md): Ai2, 2024-11-25, Dense, 7B parameters, MHA with QK-Norm
- [Llama 3.2 (1B)](https://llm-atlas-7or.pages.dev/models/llama-3-2-1b.md): Meta, 2024-09-25, Dense, 1B parameters, GQA
- [Llama 3.2 (3B)](https://llm-atlas-7or.pages.dev/models/llama-3-2-3b.md): Meta, 2024-09-25, Dense, 3B parameters, GQA
- [Llama 3 (8B)](https://llm-atlas-7or.pages.dev/models/llama-3-8b.md): Meta, 2024-04-18, Dense, 8B parameters, GQA with RoPE
- [GPT-2 XL (1.5B)](https://llm-atlas-7or.pages.dev/models/gpt-2-xl-1-5b.md): OpenAI, 2019-11-05, Dense, 1.5B parameters, MHA with learned absolute positional embeddings

## Concepts

- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Layer-wise attention budgeting (Attention budgeting)](https://llm-atlas-7or.pages.dev/concepts/attention-budgeting.md): Different layers get different attention budgets: head counts, windows or full context.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [IndexShare (IndexShare)](https://llm-atlas-7or.pages.dev/concepts/indexshare.md): Nearby layers reuse the token selection computed by one DSA indexer.
- [Qwen Sparse Attention (QSA)](https://llm-atlas-7or.pages.dev/concepts/qwen-sparse-attention.md): Like DSA, but the indexer picks whole four-token blocks rather than single tokens.
- [Compressed Sparse & Heavily Compressed Attention (CSA / HCA)](https://llm-atlas-7or.pages.dev/concepts/csa-hca.md): Pool groups of older tokens into fewer KV entries, compressing along the sequence axis.
- [Compressed Convolutional Attention (CCA)](https://llm-atlas-7or.pages.dev/concepts/compressed-convolutional-attention.md): Attention computed directly in a compressed latent space, with convolutional mixing.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Latent MoE (Latent MoE)](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md): Routed experts operate in a smaller latent space instead of the full hidden width.
- [PolyNorm (PolyNorm)](https://llm-atlas-7or.pages.dev/concepts/polynorm.md): A learned activation mixing normalised linear, quadratic and cubic terms.
- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Gated Residuals (GR)](https://llm-atlas-7or.pages.dev/concepts/gated-residuals.md): Gated read/write modules over four residual streams, without mHC's mixing matrix.
- [Attention Residuals (AttnRes)](https://llm-atlas-7or.pages.dev/concepts/attention-residuals.md): Learned, attention-style weights over earlier layer outputs replace the plain residual sum.
- [Looped Transformer (Looped depth)](https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing.md): The same blocks are applied several times, reusing weights across depth.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Cross-Layer KV Sharing (KV sharing)](https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md): Later layers reuse key/value states produced by earlier layers.
- [Per-Layer Embeddings (PLE)](https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings.md): Each layer gets its own token-specific embedding signal from a large lookup table.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Optional

- [Original gallery](https://sebastianraschka.com/llm-architecture-gallery/)
- [Report a data issue](https://github.com/rasbt/LLMs-from-scratch/issues/new?labels=architecture-gallery&title=Architecture%20Gallery%3A%20)