{
  "count": 25,
  "concepts": [
    {
      "slug": "mha",
      "name": "Multi-Head Attention",
      "short": "MHA",
      "category": "Attention",
      "tagline": "The original transformer attention: every head has its own queries, keys and values.",
      "body": [
        "Self-attention lets every token build its representation from a weighted mix of all earlier tokens. The weights come from comparing a token's query with every key; the values are what gets mixed.",
        "Multi-head attention runs several of these attention operations side by side, each in a smaller subspace, so different heads can track different relationships (syntax, coreference, position) at once. Their outputs are concatenated and projected back.",
        "In a decoder, a causal mask hides future tokens. Because every head stores its own keys and values, MHA has the largest KV cache of the common variants."
      ],
      "tradeoff": "Maximum expressiveness per layer, but the KV cache grows with heads × layers × tokens.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/mha-motivation-translation.webp",
        "alt": "Sentence translation example motivating attention",
        "width": 3566,
        "height": 2495
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/mha/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/mha.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/mha/",
      "models": [
        "olmo-3-7b",
        "ouro-thinking-2-6b",
        "olmo-2-7b",
        "gpt-2-xl-1-5b"
      ]
    },
    {
      "slug": "gqa",
      "name": "Grouped-Query Attention",
      "short": "GQA",
      "category": "Attention",
      "tagline": "Several query heads share one key/value head, shrinking the KV cache with little quality loss.",
      "body": [
        "GQA keeps many query heads but groups them so each group reads from a single shared key/value head. With 32 query heads and 8 KV heads, the cache is four times smaller than MHA.",
        "Multi-query attention (MQA) is the extreme case with one KV head for everyone; GQA sits between MQA and MHA and became the default for dense open models from Llama 2 onwards.",
        "It is simple to implement, plays well with existing kernels, and is often combined with sliding-window layers for further savings."
      ],
      "tradeoff": "Large memory and bandwidth savings at inference for a small modelling cost.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/gqa-mha-vs-gqa.webp",
        "alt": "Comparison between multi-head attention and grouped-query attention",
        "width": 1023,
        "height": 474
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/gqa/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/gqa.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/gqa/",
      "models": [
        "lfm3-1-7b",
        "minimax-m3-1-flash-428b",
        "recursive-flint-40b-a2-5b",
        "reflection-beam-501b",
        "dory1-0-4b",
        "muse-spark-open-816b-a42b",
        "nemotron-3-5-super-vl-120b-a12b",
        "kolibri-1-78b",
        "naive-n0-5-flash-309b",
        "xiaomi-mimo-v2-6-flash-rl-309b",
        "xiaomi-mimo-v2-6-pro-rl-1-02t",
        "step-5-preview-600b",
        "qwen3-8-flash-next-125b-a6b",
        "ornith-1-5-35b-a3b",
        "qwen3-8-27b",
        "nemotron-3-5-lightning-30b-a3b",
        "muse-glimmer-30b",
        "solar-open-2",
        "antares-1b",
        "laguna-s-2-1",
        "nanbeige-4-2-3b",
        "btl-3-27b",
        "inkling",
        "soofi-s-30b-a3b",
        "laguna-xs-2-1",
        "vibethinker-3b",
        "minimax-m3-428b",
        "north-mini-code-30b-a3b",
        "nemotron-3-ultra-550b-a55b",
        "gemma-4-12b",
        "jetbrains-mellum2-thinking-12b-a2-5b",
        "lfm2-5-1-2b",
        "lfm2-5-350m",
        "lfm2-5-8b-a1b",
        "command-a-218b-a25b",
        "zaya1-8b",
        "granite-4-1-30b",
        "laguna-xs-2",
        "tencent-hy3-preview-295b-a21b",
        "xiaomi-mimo-v2-5-pro-1-02t",
        "gemma-4-31b",
        "gemma-4-26b-a4b",
        "gemma-4-e4b",
        "minimax-m2-7-230b",
        "nemotron-3-nano-4b",
        "nemotron-3-super-120b-a12b",
        "sarvam-30b",
        "sarvam-105b",
        "tiny-aya-3-35b",
        "minimax-m2-5-230b",
        "nanbeige-4-1-3b",
        "step-3-5-flash-196b",
        "arcee-ai-trinity-large-400b",
        "glm-4-7-355b",
        "nemotron-3-nano-30b-a3b",
        "intellect-3",
        "olmo-3-32b",
        "minimax-m2-230b",
        "grok-2-5-270b",
        "gpt-oss-120b",
        "gpt-oss-20b",
        "qwen3-coder-flash-30b-a3b",
        "glm-4-5-355b",
        "glm-4-5-air",
        "smollm3-3b",
        "qwen3-0-6b",
        "qwen3-235b-a22b",
        "qwen3-30b-a3b",
        "qwen3-32b",
        "qwen3-4b",
        "qwen3-8b",
        "llama-4-maverick",
        "mistral-small-3-1-24b",
        "gemma-3-27b",
        "phi-4",
        "llama-3-2-1b",
        "llama-3-2-3b",
        "llama-3-8b"
      ]
    },
    {
      "slug": "mla",
      "name": "Multi-Head Latent Attention",
      "short": "MLA",
      "category": "Attention",
      "tagline": "Keys and values are compressed into a small latent vector that is cached instead of full K/V.",
      "body": [
        "Instead of sharing KV heads (as GQA does), MLA down-projects each token's keys and values into a compact latent and stores only that latent in the cache. At attention time it is projected back up.",
        "Introduced with DeepSeek V2 and popularised by DeepSeek V3/R1, MLA reported quality on par with or better than MHA while caching far less per token. A decoupled RoPE path handles positional information.",
        "Many large MoE models released since 2025 (Kimi, GLM-5, Mistral Large 3 and others) adopted MLA or variants of it."
      ],
      "tradeoff": "Very small KV cache, at the price of extra projections and a more complex implementation.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/mla-vs-mha.webp",
        "alt": "Comparison between multi-head latent attention and regular multi-head attention",
        "width": 1550,
        "height": 858
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/mla/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/mla.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/mla/",
      "models": [
        "mistral-large-4",
        "ling-3-1-flash",
        "ember-1",
        "tencent-hy4-preview-770b-a49b",
        "glm-5-3-flash",
        "ling-3-0-flash",
        "kimi-k3",
        "motif-3-beta",
        "glm-5-2",
        "kimi-k2-7-code",
        "deepseek-v4-flash",
        "deepseek-v4-pro",
        "ling-2-6-1t",
        "kimi-k2-6",
        "glm-5-1",
        "mistral-small-4",
        "sarvam-105b",
        "ling-2-5-1t",
        "glm-5-744b",
        "longcat-flash-lite-68-5b-a3b",
        "kimi-k2-5",
        "glm-4-7-355b",
        "mistral-large-3",
        "deepseek-v3-2",
        "kimi-linear-48b-a3b",
        "kimi-k2",
        "deepseek-r1",
        "deepseek-v3"
      ]
    },
    {
      "slug": "swa",
      "name": "Sliding Window Attention",
      "short": "SWA",
      "category": "Attention",
      "tagline": "Some layers only attend to a local window of recent tokens instead of the whole context.",
      "body": [
        "Local (sliding-window) layers restrict each token to the last N tokens, so their KV cache stops growing once the window is full. A few global layers are kept so information can still travel across the whole context.",
        "Gemma 3 made the pattern famous with a 5:1 local-to-global ratio and a 1,024-token window. The ratio and window size are the main knobs, and it is very often paired with GQA."
      ],
      "tradeoff": "Cheap long contexts; the risk is weaker long-range recall if too few global layers remain.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/swa-global-vs-local.webp",
        "alt": "Comparison between global attention and sliding-window attention",
        "width": 1292,
        "height": 704
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/swa/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/swa.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/swa/",
      "models": [
        "lfm3-1-7b",
        "recursive-flint-40b-a2-5b",
        "reflection-beam-501b",
        "dory1-0-4b",
        "muse-spark-open-816b-a42b",
        "kolibri-1-78b",
        "naive-n0-5-flash-309b",
        "xiaomi-mimo-v2-6-flash-rl-309b",
        "xiaomi-mimo-v2-6-pro-rl-1-02t",
        "step-5-preview-600b",
        "deepseek-v4-1-flash",
        "muse-glimmer-30b",
        "laguna-s-2-1",
        "motif-3-beta",
        "inkling",
        "laguna-xs-2-1",
        "north-mini-code-30b-a3b",
        "gemma-4-12b",
        "jetbrains-mellum2-thinking-12b-a2-5b",
        "laguna-xs-2",
        "xiaomi-mimo-v2-5-310b",
        "xiaomi-mimo-v2-5-pro-1-02t",
        "gemma-4-31b",
        "gemma-4-26b-a4b",
        "gemma-4-e2b",
        "gemma-4-e4b",
        "tiny-aya-3-35b",
        "minimax-m2-5-230b",
        "step-3-5-flash-196b",
        "arcee-ai-trinity-large-400b",
        "xiaomi-mimo-v2-flash-309b",
        "olmo-3-32b",
        "olmo-3-7b",
        "gemma-3-270m",
        "gpt-oss-120b",
        "gpt-oss-20b",
        "mistral-small-3-1-24b",
        "gemma-3-27b"
      ]
    },
    {
      "slug": "attention-budgeting",
      "name": "Layer-wise attention budgeting",
      "short": "Attention budgeting",
      "category": "Attention",
      "tagline": "Different layers get different attention budgets: head counts, windows or full context.",
      "body": [
        "Rather than giving every layer an identical attention setup, the model spends more attention capacity in some layers and less in others.",
        "Laguna XS.2 is the reference example: most of its 40 layers use a 512-token sliding window, while a minority keep full attention, and the head counts differ between the two kinds of layer."
      ],
      "tradeoff": "Puts expensive global attention only where it earns its keep, cutting KV cache and compute.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/laguna-xs2-architecture.webp",
        "alt": "Laguna XS.2 architecture",
        "width": 1258,
        "height": 1148
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/attention-budgeting/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/attention-budgeting.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/attention-budgeting/",
      "models": [
        "laguna-s-2-1",
        "laguna-xs-2-1",
        "laguna-xs-2"
      ]
    },
    {
      "slug": "gated-attention",
      "name": "Gated Attention",
      "short": "Gated attention",
      "category": "Attention",
      "tagline": "A learned sigmoid gate scales the attention output before it re-enters the residual stream.",
      "body": [
        "After the softmax attention is computed, a gate derived from the layer input multiplies the result element-wise. The gate can damp heads or channels that are not useful for the current token.",
        "Qwen3-Next and Qwen3.5 use it in their full-attention layers next to linear-attention layers; Trinity Large uses it in an attention-only stack. Reported benefits include better training stability and fewer attention-sink artefacts."
      ],
      "tradeoff": "A cheap extra projection that adds control and stability to the attention output.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/gated-attention-trinity-large.webp",
        "alt": "Trinity Large architecture and gated attention code snippet",
        "width": 3973,
        "height": 5350
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/gated-attention/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/gated-attention.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/gated-attention/",
      "models": [
        "lfm3-1-7b",
        "reflection-beam-501b",
        "dory1-0-4b",
        "muse-spark-open-816b-a42b",
        "ling-3-1-flash",
        "ember-1",
        "step-5-preview-600b",
        "tencent-hy4-preview-770b-a49b",
        "qwen3-8-flash-next-125b-a6b",
        "ornith-1-5-35b-a3b",
        "qwen3-8-27b",
        "muse-glimmer-30b",
        "ling-3-0-flash",
        "kimi-k3",
        "solar-open-2",
        "btl-3-27b",
        "qwen3-6-27b",
        "qwen3-6-35b-a3b",
        "qwen3-5-397b",
        "arcee-ai-trinity-large-400b",
        "qwen3-next-80b-a3b"
      ]
    },
    {
      "slug": "hybrid-attention",
      "name": "Hybrid Attention",
      "short": "Hybrid attention",
      "category": "Attention",
      "tagline": "Mixes cheap linear-attention or state-space layers with a few full-attention layers.",
      "body": [
        "Linear attention variants (Gated DeltaNet, Kimi Delta Attention, Lightning Attention) and state-space layers such as Mamba-2 keep a fixed-size recurrent state instead of a per-token KV cache, so their memory does not grow with context.",
        "Because they are weaker at precise retrieval, hybrids interleave them with a smaller number of full-attention layers, for example three linear layers per full-attention layer in the Qwen pattern."
      ],
      "tradeoff": "Much lower long-context memory and compute, trading away some exact recall in the linear layers.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/gated-deltanet-hybrid-overview.webp",
        "alt": "Overview of linear-attention hybrid architectures",
        "width": 3252,
        "height": 2158
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/hybrid-attention/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/hybrid-attention/",
      "models": [
        "nemotron-3-5-super-vl-120b-a12b",
        "ling-3-1-flash",
        "ember-1",
        "glm-5-3-flash",
        "qwen3-8-flash-next-125b-a6b",
        "ornith-1-5-35b-a3b",
        "qwen3-8-27b",
        "nemotron-3-5-lightning-30b-a3b",
        "ling-3-0-flash",
        "kimi-k3",
        "solar-open-2",
        "btl-3-27b",
        "soofi-s-30b-a3b",
        "nemotron-3-ultra-550b-a55b",
        "ling-2-6-1t",
        "qwen3-6-27b",
        "qwen3-6-35b-a3b",
        "nemotron-3-nano-4b",
        "nemotron-3-super-120b-a12b",
        "qwen3-5-397b",
        "ling-2-5-1t",
        "nemotron-3-nano-30b-a3b",
        "kimi-linear-48b-a3b",
        "qwen3-next-80b-a3b"
      ]
    },
    {
      "slug": "shortconv",
      "name": "Short Convolution",
      "short": "ShortConv",
      "category": "Attention",
      "tagline": "A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.",
      "body": [
        "Each channel is convolved over a handful of recent positions with its own small kernel, and future positions are never seen. It is extremely cheap and gives layers a built-in notion of local order.",
        "Kimi Linear applies ShortConv to the queries, keys and values in its delta-attention layers; Inkling places several ShortConv modules in every block. LFM-style models use convolutions as the main token mixer."
      ],
      "tradeoff": "Near-free local mixing that complements linear attention, but it cannot see far by itself.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/shortconv-kernel4-comparison.webp",
        "alt": "Kernel-size-4 causal depthwise convolution",
        "width": 1600,
        "height": 760
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/shortconv/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/shortconv.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/shortconv/",
      "models": [
        "lfm3-1-7b",
        "ling-3-1-flash",
        "ember-1",
        "glm-5-3-flash",
        "ornith-1-5-35b-a3b",
        "ling-3-0-flash",
        "kimi-k3",
        "solar-open-2",
        "inkling",
        "kimi-linear-48b-a3b"
      ]
    },
    {
      "slug": "deepseek-sparse-attention",
      "name": "DeepSeek Sparse Attention",
      "short": "DSA",
      "category": "Sparse & compressed attention",
      "tagline": "A lightweight indexer scores past tokens and each query attends only to its top-k picks.",
      "body": [
        "DSA adds a small, fast indexer that estimates how relevant each earlier token is to the current query. A selector keeps the top-k tokens (2,048 in DeepSeek's code), and full attention is computed only over those.",
        "Unlike a fixed sliding window, the selection is learned and can reach anywhere in the context. It was introduced with DeepSeek V3.2 on top of MLA and later adopted by GLM-5."
      ],
      "tradeoff": "Long-context attention cost grows with k instead of the full sequence length.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/deepseek-sparse-attention-glm5-vs-glm45.webp",
        "alt": "GLM-5 versus GLM-4.5 architecture comparison showing DeepSeek Sparse Attention",
        "width": 5807,
        "height": 2924
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/deepseek-sparse-attention/",
      "models": [
        "mistral-large-4",
        "naive-n0-5-flash-309b",
        "tencent-hy4-preview-770b-a49b",
        "glm-5-3-flash",
        "glm-5-2",
        "deepseek-v4-flash",
        "deepseek-v4-pro",
        "glm-5-1",
        "glm-5-744b",
        "deepseek-v3-2"
      ]
    },
    {
      "slug": "indexshare",
      "name": "IndexShare",
      "short": "IndexShare",
      "category": "Sparse & compressed attention",
      "tagline": "Nearby layers reuse the token selection computed by one DSA indexer.",
      "body": [
        "Computing a DSA indexer in every layer still costs time. IndexShare computes the top-k selection once and lets the following layers (three, in GLM-5.2) reuse it.",
        "The goal is to keep most of the quality of per-layer selection while cutting the indexer overhead."
      ],
      "tradeoff": "Fewer indexer passes; the selection is slightly less tailored to each layer.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/indexshare-four-layer-reuse.webp",
        "alt": "IndexShare computes top-k indices once and reuses them in following layers",
        "width": 1600,
        "height": 900
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/indexshare/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/indexshare.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/indexshare/",
      "models": [
        "mistral-large-4",
        "tencent-hy4-preview-770b-a49b",
        "glm-5-2"
      ]
    },
    {
      "slug": "qwen-sparse-attention",
      "name": "Qwen Sparse Attention",
      "short": "QSA",
      "category": "Sparse & compressed attention",
      "tagline": "Like DSA, but the indexer picks whole four-token blocks rather than single tokens.",
      "body": [
        "The indexer averages keys within each complete four-token block, scores the pooled keys against the query, and expands the winning blocks back into their tokens. The current incomplete block is always visible.",
        "Selecting contiguous blocks is friendlier to memory access patterns than scattering over individual token positions."
      ],
      "tradeoff": "Hardware-friendlier sparse reads at a coarser selection granularity.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-8-flash-next.webp",
        "alt": "Qwen3.8-Flash-Next architecture showing Qwen Sparse Attention",
        "width": 4077,
        "height": 3390
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/qwen-sparse-attention/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/qwen-sparse-attention.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/qwen-sparse-attention/",
      "models": [
        "qwen3-8-flash-next-125b-a6b"
      ]
    },
    {
      "slug": "csa-hca",
      "name": "Compressed Sparse & Heavily Compressed Attention",
      "short": "CSA / HCA",
      "category": "Sparse & compressed attention",
      "tagline": "Pool groups of older tokens into fewer KV entries, compressing along the sequence axis.",
      "body": [
        "MLA shrinks each token's cache entry; CSA and HCA reduce how many entries there are. Groups of past tokens are pooled into compressed entries.",
        "CSA compresses moderately and reads a sparse selection of entries; HCA compresses aggressively and reads all of them. DeepSeek V4 combines both to reach million-token contexts."
      ],
      "tradeoff": "Drastically smaller long-context caches, with some loss of token-level detail in the far past.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/deepseek-v4-csa-hca-cache-comparison.webp",
        "alt": "MLA-style cache, CSA, and HCA compared",
        "width": 1990,
        "height": 664
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/csa-hca/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/csa-hca.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/csa-hca/",
      "models": [
        "deepseek-v4-flash",
        "deepseek-v4-pro"
      ]
    },
    {
      "slug": "compressed-convolutional-attention",
      "name": "Compressed Convolutional Attention",
      "short": "CCA",
      "category": "Sparse & compressed attention",
      "tagline": "Attention computed directly in a compressed latent space, with convolutional mixing.",
      "body": [
        "Where MLA uses a latent mainly as a compact cache format and decompresses it for attention, CCA performs the attention operation itself in the compressed space and adds convolutions to mix information.",
        "ZAYA1-8B by Zyphra uses CCA together with grouped-query attention."
      ],
      "tradeoff": "Saves both cache memory and attention FLOPs; still a young technique.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/zaya1-cca-architecture.webp",
        "alt": "ZAYA1-8B architecture with compressed convolutional attention",
        "width": 1354,
        "height": 1384
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/compressed-convolutional-attention/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/compressed-convolutional-attention.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/compressed-convolutional-attention/",
      "models": [
        "zaya1-8b"
      ]
    },
    {
      "slug": "moe",
      "name": "Mixture of Experts",
      "short": "MoE",
      "category": "Feed-forward & experts",
      "tagline": "Many feed-forward experts, but a router sends each token to only a few of them.",
      "body": [
        "The single feed-forward block is replaced by many expert blocks. A router picks a small number (often 8) per token, so the model has a huge total parameter count while each token only touches a fraction of them, the active parameters.",
        "Many designs add a shared expert that every token uses, and keep the first few layers dense. DeepSeek V3 is the template much of the 2025–2026 wave followed."
      ],
      "tradeoff": "More knowledge capacity per unit of compute, but all experts must still fit in memory.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/moe-deepseek-v3-vs-ffn.webp",
        "alt": "Mixture-of-Experts module compared with a standard feed-forward block",
        "width": 1304,
        "height": 822
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/moe/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/moe.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/moe/",
      "models": [
        "mistral-large-4",
        "minimax-m3-1-flash-428b",
        "recursive-flint-40b-a2-5b",
        "reflection-beam-501b",
        "muse-spark-open-816b-a42b",
        "nemotron-3-5-super-vl-120b-a12b",
        "ling-3-1-flash",
        "kolibri-1-78b",
        "naive-n0-5-flash-309b",
        "ember-1",
        "xiaomi-mimo-v2-6-flash-rl-309b",
        "xiaomi-mimo-v2-6-pro-rl-1-02t",
        "step-5-preview-600b",
        "deepseek-v4-1-flash",
        "tencent-hy4-preview-770b-a49b",
        "glm-5-3-flash",
        "qwen3-8-flash-next-125b-a6b",
        "ornith-1-5-35b-a3b",
        "nemotron-3-5-lightning-30b-a3b",
        "ling-3-0-flash",
        "kimi-k3",
        "solar-open-2",
        "laguna-s-2-1",
        "motif-3-beta",
        "inkling",
        "soofi-s-30b-a3b",
        "laguna-xs-2-1",
        "glm-5-2",
        "minimax-m3-428b",
        "kimi-k2-7-code",
        "north-mini-code-30b-a3b",
        "nemotron-3-ultra-550b-a55b",
        "jetbrains-mellum2-thinking-12b-a2-5b",
        "lfm2-5-8b-a1b",
        "command-a-218b-a25b",
        "zaya1-8b",
        "laguna-xs-2",
        "deepseek-v4-flash",
        "deepseek-v4-pro",
        "tencent-hy3-preview-295b-a21b",
        "qwen3-6-27b",
        "xiaomi-mimo-v2-5-310b",
        "xiaomi-mimo-v2-5-pro-1-02t",
        "kimi-k2-6",
        "qwen3-6-35b-a3b",
        "glm-5-1",
        "gemma-4-26b-a4b",
        "minimax-m2-7-230b",
        "mistral-small-4",
        "nemotron-3-super-120b-a12b",
        "sarvam-30b",
        "sarvam-105b",
        "qwen3-5-397b",
        "minimax-m2-5-230b",
        "glm-5-744b",
        "step-3-5-flash-196b",
        "longcat-flash-lite-68-5b-a3b",
        "arcee-ai-trinity-large-400b",
        "kimi-k2-5",
        "glm-4-7-355b",
        "xiaomi-mimo-v2-flash-309b",
        "nemotron-3-nano-30b-a3b",
        "mistral-large-3",
        "deepseek-v3-2",
        "intellect-3",
        "minimax-m2-230b",
        "qwen3-next-80b-a3b",
        "grok-2-5-270b",
        "gpt-oss-120b",
        "gpt-oss-20b",
        "qwen3-coder-flash-30b-a3b",
        "glm-4-5-355b",
        "glm-4-5-air",
        "kimi-k2",
        "qwen3-235b-a22b",
        "qwen3-30b-a3b",
        "llama-4-maverick",
        "deepseek-r1",
        "deepseek-v3"
      ]
    },
    {
      "slug": "latent-moe",
      "name": "Latent MoE",
      "short": "Latent MoE",
      "category": "Feed-forward & experts",
      "tagline": "Routed experts operate in a smaller latent space instead of the full hidden width.",
      "body": [
        "Tokens are projected down before they enter the routed experts and projected back up afterwards. Experts become cheaper, so a model can afford more of them or activate more per token.",
        "Nemotron 3 Super and Ultra, Kimi K3 and Muse Spark Open use this layout with different compression ratios and normalization details."
      ],
      "tradeoff": "Cheaper experts and communication; the bottleneck must be sized carefully.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/latent-moe-nemotron-super.webp",
        "alt": "Nemotron 3 Super architecture showing latent MoE layers",
        "width": 1862,
        "height": 1656
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/latent-moe/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/latent-moe.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/latent-moe/",
      "models": [
        "muse-spark-open-816b-a42b",
        "nemotron-3-5-super-vl-120b-a12b",
        "ember-1",
        "kimi-k3",
        "nemotron-3-super-120b-a12b"
      ]
    },
    {
      "slug": "polynorm",
      "name": "PolyNorm",
      "short": "PolyNorm",
      "category": "Feed-forward & experts",
      "tagline": "A learned activation mixing normalised linear, quadratic and cubic terms.",
      "body": [
        "Feed-forward blocks moved from GELU to SwiGLU-style gating with a fixed SiLU. PolyNorm replaces that fixed function with a learned combination of separately RMS-normalised powers of the pre-activation.",
        "Motif 3 Beta uses a grouped PolyNorm inside each expert."
      ],
      "tradeoff": "A more flexible nonlinearity for a little extra compute and tuning.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/polynorm-gated-ffn.webp",
        "alt": "SwiGLU compared with PolyNorm",
        "width": 1600,
        "height": 900
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/polynorm/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/polynorm.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/polynorm/",
      "models": [
        "motif-3-beta"
      ]
    },
    {
      "slug": "mhc",
      "name": "Manifold-constrained Hyper-Connections",
      "short": "mHC",
      "category": "Residual stream",
      "tagline": "Several parallel residual streams with learned, constrained mixing between them.",
      "body": [
        "The single residual stream is widened into several streams (four in DeepSeek V4). Learned mappings decide how sublayer inputs are read from, and outputs written back to, these streams.",
        "Unconstrained hyper-connections can amplify signals and destabilise training; mHC constrains the mixing matrix to a well-behaved manifold so information is mixed without blowing up. DeepSeek V4.1 introduced a cheaper single-pass variant."
      ],
      "tradeoff": "A richer residual path for a modest training overhead.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/mhc-hyper-connections.webp",
        "alt": "Regular transformer block compared with hyper-connections",
        "width": 1232,
        "height": 1528
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/mhc/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/mhc.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/mhc/",
      "models": [
        "deepseek-v4-1-flash",
        "tencent-hy4-preview-770b-a49b",
        "glm-5-3-flash",
        "motif-3-beta",
        "deepseek-v4-flash",
        "deepseek-v4-pro"
      ]
    },
    {
      "slug": "gated-residuals",
      "name": "Gated Residuals",
      "short": "GR",
      "category": "Residual stream",
      "tagline": "Gated read/write modules over four residual streams, without mHC's mixing matrix.",
      "body": [
        "Before a sublayer, a GR Read module normalises each of four residual streams, gates them element-wise and averages them into a normal-width input. Afterwards, GR Write adds the output back to all four streams with learned gates.",
        "It was introduced with Qwen3.8-Flash-Next and is related in spirit to mHC."
      ],
      "tradeoff": "Wider residual capacity with simple gating instead of a constrained mixing matrix.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-8-flash-next.webp",
        "alt": "Qwen3.8-Flash-Next architecture showing GR Read and GR Write",
        "width": 4077,
        "height": 3390
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/gated-residuals/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/gated-residuals.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/gated-residuals/",
      "models": [
        "qwen3-8-flash-next-125b-a6b"
      ]
    },
    {
      "slug": "attention-residuals",
      "name": "Attention Residuals",
      "short": "AttnRes",
      "category": "Residual stream",
      "tagline": "Learned, attention-style weights over earlier layer outputs replace the plain residual sum.",
      "body": [
        "A standard residual connection adds every earlier update with weight one. AttnRes lets each layer attend over earlier layer outputs and weight their contributions.",
        "Full and block variants exist; Kimi Linear already used the idea. Reported gains are consistent but modest, for a few percent extra training and inference cost."
      ],
      "tradeoff": "Smarter depth-wise information flow for a small overhead.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/attention-residuals-overview.webp",
        "alt": "Standard residuals compared with Attention Residuals",
        "width": 2122,
        "height": 1198
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/attention-residuals/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/attention-residuals.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/attention-residuals/",
      "models": [
        "ember-1",
        "kimi-k3",
        "kimi-linear-48b-a3b"
      ]
    },
    {
      "slug": "looped-depth-sharing",
      "name": "Looped Transformer",
      "short": "Looped depth",
      "category": "Residual stream",
      "tagline": "The same blocks are applied several times, reusing weights across depth.",
      "body": [
        "With L distinct blocks applied T times, the model gets L × T effective layers for the parameter cost of L. It is depth-wise weight tying, sometimes called recurrent depth.",
        "Nanbeige 4.2 uses two fixed passes, Ouro uses four passes with a learned exit gate, and Dory1.0 loops a shared middle stack."
      ],
      "tradeoff": "More compute per parameter; extra passes add latency and, depending on the design, KV cache.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nanbeige-4-2-3b.webp",
        "alt": "Nanbeige 4.2 architecture with a loop around the shared transformer stack",
        "width": 2683,
        "height": 2663
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing/",
      "models": [
        "dory1-0-4b",
        "nanbeige-4-2-3b",
        "ouro-thinking-2-6b"
      ]
    },
    {
      "slug": "qk-norm",
      "name": "QK-Norm",
      "short": "QK-Norm",
      "category": "Positions & normalization",
      "tagline": "RMSNorm applied to queries and keys before attention to keep logits well-behaved.",
      "body": [
        "Normalising queries and keys (usually before RoPE) prevents attention logits from growing too large, a common source of loss spikes in large-scale training.",
        "OLMo 2 and Qwen3 helped make it standard; it is now a default ingredient in many recipes."
      ],
      "tradeoff": "A stabiliser rather than an efficiency trick, and it costs almost nothing.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/qk-norm-training-stability.webp",
        "alt": "Training stability comparison associated with OLMo 2's normalization changes",
        "width": 1346,
        "height": 407
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/qk-norm/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/qk-norm.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/qk-norm/",
      "models": [
        "lfm3-1-7b",
        "minimax-m3-1-flash-428b",
        "recursive-flint-40b-a2-5b",
        "reflection-beam-501b",
        "dory1-0-4b",
        "muse-spark-open-816b-a42b",
        "kolibri-1-78b",
        "step-5-preview-600b",
        "qwen3-8-flash-next-125b-a6b",
        "ornith-1-5-35b-a3b",
        "qwen3-8-27b",
        "muse-glimmer-30b",
        "laguna-s-2-1",
        "btl-3-27b",
        "inkling",
        "laguna-xs-2-1",
        "minimax-m3-428b",
        "gemma-4-12b",
        "laguna-xs-2",
        "tencent-hy3-preview-295b-a21b",
        "gemma-4-31b",
        "gemma-4-26b-a4b",
        "gemma-4-e2b",
        "gemma-4-e4b",
        "minimax-m2-7-230b",
        "sarvam-30b",
        "minimax-m2-5-230b",
        "arcee-ai-trinity-large-400b",
        "glm-4-7-355b",
        "olmo-3-32b",
        "olmo-3-7b",
        "minimax-m2-230b",
        "gemma-3-270m",
        "glm-4-5-355b",
        "qwen3-235b-a22b",
        "qwen3-32b",
        "qwen3-4b",
        "qwen3-8b",
        "gemma-3-27b",
        "olmo-2-7b"
      ]
    },
    {
      "slug": "nope",
      "name": "No Positional Embeddings",
      "short": "NoPE",
      "category": "Positions & normalization",
      "tagline": "Some or all layers drop explicit positional encodings and rely on the causal mask.",
      "body": [
        "Because a decoder's causal mask already prevents tokens from seeing the future, the model can infer order implicitly. NoPE layers omit RoPE entirely.",
        "Studies suggest this can help length generalisation. SmolLM3 removes RoPE from every fourth layer; Kimi K3 goes NoPE throughout."
      ],
      "tradeoff": "Potentially better extrapolation to longer contexts; less explicit positional signal.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/nope-length-generalization.webp",
        "alt": "Annotated NoPE figure about length generalization",
        "width": 1364,
        "height": 800
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/nope/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/nope.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/nope/",
      "models": [
        "mistral-large-4",
        "recursive-flint-40b-a2-5b",
        "reflection-beam-501b",
        "muse-spark-open-816b-a42b",
        "kolibri-1-78b",
        "ember-1",
        "glm-5-3-flash",
        "muse-glimmer-30b",
        "kimi-k3",
        "solar-open-2",
        "soofi-s-30b-a3b",
        "north-mini-code-30b-a3b",
        "command-a-218b-a25b",
        "sarvam-105b",
        "tiny-aya-3-35b",
        "arcee-ai-trinity-large-400b",
        "kimi-linear-48b-a3b",
        "smollm3-3b"
      ]
    },
    {
      "slug": "kv-sharing",
      "name": "Cross-Layer KV Sharing",
      "short": "KV sharing",
      "category": "Efficiency & sharing",
      "tagline": "Later layers reuse key/value states produced by earlier layers.",
      "body": [
        "GQA shares KV heads across query heads within a layer; cross-layer sharing goes further and lets whole layers reuse another layer's keys and values, so fewer layers add to the cache.",
        "Gemma 4 E2B and E4B are prominent examples of the idea from cross-layer attention research."
      ],
      "tradeoff": "Smaller cache and less compute for long contexts, with less per-layer flexibility.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/gemma4-kv-sharing.webp",
        "alt": "Cross-layer KV sharing in Gemma 4",
        "width": 1456,
        "height": 1181
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/kv-sharing/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/kv-sharing/",
      "models": [
        "mistral-large-4",
        "deepseek-v4-1-flash",
        "gemma-4-e2b",
        "gemma-4-e4b"
      ]
    },
    {
      "slug": "per-layer-embeddings",
      "name": "Per-Layer Embeddings",
      "short": "PLE",
      "category": "Efficiency & sharing",
      "tagline": "Each layer gets its own token-specific embedding signal from a large lookup table.",
      "body": [
        "Lookup tables are cheap to run and can live outside fast memory, so small models can store extra token-specific information per layer without making the transformer stack itself more expensive.",
        "It is why Gemma 4 E2B is described as 2.3B effective parameters but 5.1B including embeddings. The “E” stands for effective."
      ],
      "tradeoff": "More capacity for small on-device models at very low compute cost.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/gemma4-ple-residual-path.webp",
        "alt": "Simplified Gemma 4 block with the per-layer embedding residual path",
        "width": 1652,
        "height": 1420
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/per-layer-embeddings/",
      "models": [
        "qwen3-8-flash-next-125b-a6b",
        "gemma-4-e2b",
        "gemma-4-e4b"
      ]
    },
    {
      "slug": "mtp",
      "name": "Multi-Token Prediction",
      "short": "MTP",
      "category": "Training objectives",
      "tagline": "Extra heads learn to predict several future tokens, and can draft tokens at inference.",
      "body": [
        "Besides the next token, small additional modules predict tokens further ahead. DeepSeek's version keeps a causal chain where each module conditions on the previous prediction.",
        "The extra targets give a denser training signal, and at inference the MTP modules act as a built-in draft model for speculative decoding."
      ],
      "tradeoff": "Denser training signal and faster decoding, for some extra parameters.",
      "figure": {
        "src": "https://sebastianraschka.com/llm-architecture-gallery/images/concepts/mtp-next-token-vs-multi-token.webp",
        "alt": "Next-token prediction compared with multi-token prediction",
        "width": 2048,
        "height": 1284
      },
      "url": "https://llm-atlas-7or.pages.dev/concepts/mtp/",
      "markdownUrl": "https://llm-atlas-7or.pages.dev/concepts/mtp.md",
      "sourceUrl": "https://sebastianraschka.com/llm-architecture-gallery/mtp/",
      "models": [
        "nemotron-3-5-super-vl-120b-a12b",
        "ling-3-1-flash",
        "xiaomi-mimo-v2-6-flash-rl-309b",
        "xiaomi-mimo-v2-6-pro-rl-1-02t",
        "step-5-preview-600b",
        "tencent-hy4-preview-770b-a49b",
        "glm-5-3-flash",
        "qwen3-8-flash-next-125b-a6b",
        "ornith-1-5-35b-a3b",
        "qwen3-8-27b",
        "nemotron-3-5-lightning-30b-a3b",
        "ling-3-0-flash",
        "motif-3-beta",
        "inkling",
        "deepseek-v4-flash",
        "deepseek-v4-pro",
        "ling-2-6-1t",
        "tencent-hy3-preview-295b-a21b",
        "xiaomi-mimo-v2-5-310b",
        "glm-5-1",
        "minimax-m2-7-230b",
        "nemotron-3-super-120b-a12b",
        "minimax-m2-5-230b",
        "glm-5-744b",
        "step-3-5-flash-196b",
        "glm-4-7-355b",
        "xiaomi-mimo-v2-flash-309b",
        "deepseek-v3-2",
        "minimax-m2-230b",
        "qwen3-next-80b-a3b",
        "glm-4-5-355b",
        "glm-4-5-air",
        "deepseek-r1",
        "deepseek-v3"
      ]
    }
  ]
}