# LLM Atlas

> Browse, filter and compare 100+ open-weight LLM architectures: diagrams, attention mechanisms, MoE layouts, KV-cache sizes and fact sheets, sourced from Sebastian Raschka's LLM Architecture Gallery.

Source of truth: Sebastian Raschka's LLM Architecture Gallery — https://sebastianraschka.com/llm-architecture-gallery/. All architecture diagrams and model facts originate there; this site re-presents them. Data synced 2026-11-04. 118 models from 44 labs, released 2019-11-05 to 2026-10-29.

Every page is static HTML. Machine-readable alternatives: a Markdown file per model and concept (append `.md` to the page path without the trailing slash) and a JSON API (no auth, CORS-safe static files).

---

## Model: Mistral Large 4 (1T)

> Mistral's trillion-parameter multimodal MoE combines MLA with sparse token selection and cross-layer KV sharing.

| Property | Value |
| --- | --- |
| Lab | Mistral AI (France) |
| Released | 2026-10-29 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1T total, 52B active (5.2% active) |
| Context window | 1,048,576 tokens |
| Attention | MLA with modified DeepSeek Sparse Attention, index reuse, and KV sharing |
| Layer mix | 72 MLA/DSA |
| KV cache / token (bf16) | 80.75 KiB (Moderate) |
| License | Mistral Ecosystem Open-Weight License |

**Key detail:** Selects the top 2,048 tokens, reuses sparse indices across layers, and alternates three LeRoPE layers with one NoPE layer.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [IndexShare (IndexShare)](https://llm-atlas-7or.pages.dev/concepts/indexshare.md): Nearby layers reuse the token selection computed by one DSA indexer.
- [Cross-Layer KV Sharing (KV sharing)](https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md): Later layers reuse key/value states produced by earlier layers.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 26 (Knowledge Work 40.6, Coding 18.9, Science 13, General 31.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mistral-large-4.webp
- config.json: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/config.json
- License: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/LICENSE
- Tech report: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B/blob/main/assets/technical_report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-mistral-large-4
- Web page: https://llm-atlas-7or.pages.dev/models/mistral-large-4/
- JSON: https://llm-atlas-7or.pages.dev/api/models/mistral-large-4.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: LFM3 (1.7B)

> Liquid AI's 1.73B dense hybrid combines short convolutions with sliding-window and global GQA in a 24-layer stack.

| Property | Value |
| --- | --- |
| Lab | Liquid AI (United States) |
| Released | 2026-10-27 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 1.73B parameters |
| Context window | 262,144 tokens |
| Attention | ShortConv plus gated GQA with QK-Norm and sliding-window/global attention |
| Layer mix | 13 ShortConv + 5 sliding-window + 6 global GQA |
| KV cache / token (bf16) | 22 KiB (Very low) |
| License | LFM Open License v1.0 |

**Key detail:** Uses kernel-7 ShortConv, a 256-token attention window, per-head sigmoid gates, and sandwich RMSNorm. Global attention uses 50% partial RoPE.

## Architecture concepts

- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/lfm3-1-7b.webp
- config.json: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/main/config.json
- License: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/main/LICENSE
- Tech report: https://huggingface.co/blog/LiquidAI/lfm3
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-lfm3-1-7b
- Web page: https://llm-atlas-7or.pages.dev/models/lfm3-1-7b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/lfm3-1-7b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: MiniMax M3.1-Flash (428B)

> MiniMax M3 successor with sparse attention in all 60 decoder layers, quantization-aware training, and a DSpark draft head for speculative decoding.

| Property | Value |
| --- | --- |
| Lab | MiniMax (China) |
| Released | 2026-10-19 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 428B total, 23B active (5.4% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with QK-Norm and MiniMax Sparse Attention |
| Layer mix | 60 MiniMax Sparse Attention; 3 dense FFN + 57 MoE |
| KV cache / token (bf16) | 120 KiB (Moderate) |
| License | MiniMax Community License |

**Key detail:** All 60 layers use sparse attention; the first 3 retain dense FFNs, while the remaining 57 route to 4 of 128 experts plus 1 shared expert per token.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 28.9 (Knowledge Work 47.9, Coding 22.8, Science 14.4, General 30.5)
- AA Intelligence Index v4.3.2: total 42.7 (General 40.1, Scientific 35.1, Coding 39.6, Agents 52.4)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/minimax-m3.1-flash.webp
- config.json: https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/config.json
- License: https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2606.13392
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-minimax-m3-1-flash-428b
- Web page: https://llm-atlas-7or.pages.dev/models/minimax-m3-1-flash-428b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/minimax-m3-1-flash-428b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Recursive Flint (40B-A2.5B)

> Recursive's coding model uses two dense FFN layers followed by 47 MoE layers. The active-parameter count includes the output head.

| Property | Value |
| --- | --- |
| Lab | Recursive (United States) |
| Released | 2026-10-18 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 39.8B total, 2.5B active (6.3% active) |
| Context window | 262,144 configured; 512k supported tokens |
| Attention | GQA with QK-Norm, learned attention sinks, and sliding-window/global attention |
| Layer mix | 35 sliding-window + 14 global GQA |
| KV cache / token (bf16) | 98 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Uses top-6 routing over 256 experts plus one shared expert, with RoPE in local layers, NoPE in global layers, and a local window covering the current token and 512 preceding tokens.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/recursive-flint.webp
- config.json: https://huggingface.co/RecursiveSI/flint/blob/main/config.json
- Tech report: https://recursive.com/updates/introducing-recursive-flint
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-recursive-flint-40b-a2-5b
- Web page: https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/recursive-flint-40b-a2-5b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Reflection Beam (501B)

> Text-only MoE decoder with three dense FFN layers followed by 49 expert layers. Each MoE layer activates 8 of 256 experts without a shared expert.

| Property | Value |
| --- | --- |
| Lab | Reflection (United States) |
| Released | 2026-10-16 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 501.3B total, 23.4B active (4.7% active) |
| Context window | 1,048,576 tokens |
| Attention | Gated GQA with QK-Norm and sliding-window/global attention |
| Layer mix | 43 sliding-window + 9 global GQA |
| KV cache / token (bf16) | 208 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Combines 512-token local RoPE attention with global NoPE and query scaling. Uses sandwich RMSNorm, depth-scaled residuals, and a final logit softcap of 30.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 17.7 (Knowledge Work 26.3, Coding 4.5, Science 10.9, General 29.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/reflection-beam.webp
- config.json: https://huggingface.co/reflection/Beam-1.0-501B-A23B/blob/main/config.json
- License: https://huggingface.co/reflection/Beam-1.0-501B-A23B/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-reflection-beam-501b
- Web page: https://llm-atlas-7or.pages.dev/models/reflection-beam-501b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/reflection-beam-501b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Dory1.0 (4B)

> NVIDIA's dense looped transformer applies a shared 20-block core twice between 8-block input and output stacks, using normalized GPT (nGPT) residual updates.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-10-16 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 4B parameters |
| Context window | 262,144 tokens |
| Attention | Gated GQA; 3:1 sliding-window/full, RoPE |
| Layer mix | 8 input + 20 shared × 2 passes + 8 output (56 applications) |
| KV cache / token (bf16) | 224 KiB (High) |
| License | OpenMDW 1.1 |

**Key detail:** Reinjects the input-stack output before each recurrent pass. The 36 distinct blocks give 56 block applications by default, with L2-normalized residual updates and a 1,024-token attention window.

## Architecture concepts

- [Looped Transformer (Looped depth)](https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing.md): The same blocks are applied several times, reusing weights across depth.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/dory-1-0-4b.webp
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/main/config.json
- License: https://openmdw.ai/license/1-1/
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-dory1-0-4b
- Web page: https://llm-atlas-7or.pages.dev/models/dory1-0-4b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/dory1-0-4b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Muse Spark Open (816B-A42B)

> Meta's multimodal MoE model combines 62 latent-expert blocks with gated GQA, local-only RoPE, global NoPE, and learned residual gates; the figure focuses on the text decoder.

| Property | Value |
| --- | --- |
| Lab | Meta (United States) |
| Released | 2026-10-15 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 816B total, 42B active (5.1% active) |
| Context window | 1,048,576 tokens |
| Attention | Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention |
| Layer mix | 46 sliding-window + 16 global gated GQA |
| KV cache / token (bf16) | 248 KiB (High) |
| License | Muse Model License |

**Key detail:** Selects 8 of 256 experts, compresses the expert path from 8,192 to 4,096 dimensions, normalizes each expert output before combining, and uses no shared expert.

## Architecture concepts

- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Latent MoE (Latent MoE)](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md): Routed experts operate in a smaller latent space instead of the full hidden width.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 24.7 (Knowledge Work 37, Coding 7.8, Science 12.1, General 41.7)
- AA Intelligence Index v4.3.2: total 38.9 (General 45, Scientific 29.4, Coding 33.3, Agents 42.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/muse-spark-open.webp
- config.json: https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B/blob/main/config.json
- License: https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B/blob/main/LICENSE.md
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-muse-spark-open-816b-a42b
- Web page: https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/muse-spark-open-816b-a42b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nemotron 3.5 Super VL (120B-A12B)

> NVIDIA's vision-language model extends the Nemotron 3 Super backbone with C-RADIO vision encoding and shared-weight multi-token prediction. The figure shows the language backbone, with the vision encoder and MTP omitted.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-10-15 |
| Decoder type | Hybrid MoE |
| Family | Hybrid MoE |
| Scale | 120B total, 12B active (10% active) |
| Context window | 1,048,576 tokens |
| Attention | Mostly Mamba-2 with eight GQA layers |
| Layer mix | 8 GQA + 40 Mamba-2 + 40 LatentMoE; GQA + MoE MTP predictor |
| KV cache / token (bf16) | 8 KiB (Very low) |
| License | OpenMDW 1.1 |

**Key detail:** Routes each token to 22 of 512 experts in a 1,024-dimensional latent space, alongside one shared expert at the full 4,096-dimensional width.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Latent MoE (Latent MoE)](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md): Routed experts operate in a smaller latent space instead of the full hidden width.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 13.6 (Knowledge Work 19.5, Coding 3.7, Science 5.4, General 25.9)
- AA Intelligence Index v4.3.2: total 26.5 (General 32.6, Scientific 20.6, Coding 23.9, Agents 26)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nemotron-3-5-super-vl-120b-a12b.webp
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4/blob/main/config.json
- License: https://openmdw.ai/license/1-1/
- Tech report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nemotron-3-5-super-vl-120b-a12b
- Web page: https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nemotron-3-5-super-vl-120b-a12b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ling 3.1 Flash (560B)

> Hybrid MoE that extends Ling 3.0 Flash to 72 layers, with seven Kimi Delta Attention layers for every gated MLA layer.

| Property | Value |
| --- | --- |
| Lab | InclusionAI (China) |
| Released | 2026-10-13 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 560B total, 25B active (4.5% active) |
| Context window | 1M reported; 262,144 in config tokens |
| Attention | 7:1 Kimi Delta Attention and gated MLA |
| Layer mix | 63 Kimi Delta Attention + 9 gated MLA + 1 MTP layer |
| KV cache / token (bf16) | 10.1 KiB (Very low) |
| License | MIT License |

**Key detail:** Uses four initial dense FFN layers, top-8 routing over 512 experts plus one shared expert, and MLA with a 1,536-dimensional query bottleneck and 512-dimensional KV latent.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ling-3-1-flash.webp
- config.json: https://huggingface.co/inclusionAI/Ling-3.1-flash/blob/main/config.json
- Tech report: https://huggingface.co/inclusionAI/Ling-3.1-flash
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ling-3-1-flash
- Web page: https://llm-atlas-7or.pages.dev/models/ling-3-1-flash/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ling-3-1-flash.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kolibri-1 (78B)

> Aleph Alpha's German-English reasoning model uses 50 MoE blocks with a 4:1 mix of sliding-window and global GQA.

| Property | Value |
| --- | --- |
| Lab | Aleph Alpha (Germany) |
| Released | 2026-10-03 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 78B total, 3.46B active (4.4% active) |
| Context window | 262,144 native; 1,048,576 extrapolated tokens |
| Attention | GQA with QK-Norm and 4:1 sliding-window/global attention |
| Layer mix | 40 sliding-window + 10 global GQA |
| KV cache / token (bf16) | 100 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Uses top-6 routing over 384 experts plus one shared expert, with sandwich RMSNorm, RoPE in local layers, and NoPE in global layers.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kolibri-1.webp
- config.json: https://huggingface.co/Aleph-Alpha/Kolibri-1/blob/main/config.json
- Tech report: https://aleph-alpha.com/downloads/tech-report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kolibri-1-78b
- Web page: https://llm-atlas-7or.pages.dev/models/kolibri-1-78b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kolibri-1-78b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Naive-N0.5-Flash (309B)

> MiMo-V2.5-derived sparse MoE that replaces its global attention layers with GQA-based DeepSeek Sparse Attention for a native 1M-token context.

| Property | Value |
| --- | --- |
| Lab | NaiveAI (China) |
| Released | 2026-09-27 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 309B total, 15.5B active (5.0% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with sliding-window attention and DeepSeek Sparse Attention |
| Layer mix | 39 sliding-window + 9 GQA/DSA |
| KV cache / token (bf16) | 219.75 KiB (High) |
| License | MIT License |

**Key detail:** Pairs a 128-token local window with top-2,048 DSA selection, four DSA KV heads, and a 16-head indexer. Uses 256 routed experts with top-8 routing and no full-attention layers.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/naive-n0-5-flash-309b.webp
- config.json: https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/config.json
- License: https://huggingface.co/NaiveAI/Naive-N0.5-Flash/blob/main/LICENSE
- Tech report: https://naive.ai/en/research/
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-naive-n0-5-flash-309b
- Web page: https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/naive-n0-5-flash-309b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ember-1 (2.8T)

> Fireworks' Kimi K3 derivative, post-trained for shorter reasoning traces. Architecture details follow the published Kimi K3 base model; the diagram focuses on its text backbone.

| Property | Value |
| --- | --- |
| Lab | Fireworks AI (United States) |
| Released | 2026-09-23 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 2.8T total, 104B active (3.7% active) |
| Context window | 1,048,576 tokens |
| Attention | 3:1 Kimi Delta Attention and Gated MLA with NoPE |
| Layer mix | 69 KDA + 24 Gated MLA |
| KV cache / token (bf16) | 27 KiB (Low) |
| License | N/A |

**Key detail:** Inherits Kimi K3's eight Block Attention Residual groups and Stable LatentMoE, with one dense layer followed by 92 MoE layers.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Attention Residuals (AttnRes)](https://llm-atlas-7or.pages.dev/concepts/attention-residuals.md): Learned, attention-style weights over earlier layer outputs replace the plain residual sum.
- [Latent MoE (Latent MoE)](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md): Routed experts operate in a smaller latent space instead of the full hidden width.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ember-1.webp
- Kimi K3 base config: https://huggingface.co/moonshotai/Kimi-K3/blob/main/config.json
- Tech report: https://fireworks.ai/blog/ember-1
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ember-1
- Web page: https://llm-atlas-7or.pages.dev/models/ember-1/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ember-1.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Xiaomi MiMo-V2.6-Flash-RL (309B)

> Omnimodal sparse MoE model with the MiMo-V2.5 text backbone, mixed-domain reinforcement learning, and a five-layer DFlash speculative decoder.

| Property | Value |
| --- | --- |
| Lab | Xiaomi (China) |
| Released | 2026-09-21 |
| Decoder type | Sparse omnimodal MoE |
| Family | Sparse MoE |
| Scale | 309B total, 15B active (4.9% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with sliding-window/global attention |
| Layer mix | 39 sliding-window + 9 global |
| KV cache / token (bf16) | 217.5 KiB (High) |
| License | MIT License |

**Key detail:** Uses 256 routed experts with top-8 routing, one dense prefix layer, no shared experts, and a 128-token sliding window.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 26.7 (Knowledge Work 44.2, Coding 22.1, Science 14.4, General 25.9)
- AA Intelligence Index v4.3.2: total 37.9 (General 32, Scientific 23.5, Coding 37, Agents 53.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mimo-2.6-flash.webp
- config.json: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/main/config.json
- Tech report: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/resolve/main/MiMo_V2_6_technical_report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-xiaomi-mimo-v2-6-flash-rl-309b
- Web page: https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/xiaomi-mimo-v2-6-flash-rl-309b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Xiaomi MiMo-V2.6-Pro-RL (1.02T)

> Omnimodal sparse MoE model with the MiMo-V2.5-Pro text backbone, mixed-domain reinforcement learning, and a five-layer DFlash speculative decoder.

| Property | Value |
| --- | --- |
| Lab | Xiaomi (China) |
| Released | 2026-09-21 |
| Decoder type | Sparse omnimodal MoE |
| Family | Sparse MoE |
| Scale | 1.02T total, 42B active (4.1% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with 6:1 sliding-window/global attention |
| Layer mix | 60 sliding-window + 10 global |
| KV cache / token (bf16) | 350 KiB (Very high) |
| License | MIT License |

**Key detail:** Uses 384 routed experts with top-8 routing, one dense prefix layer, no shared experts, and a 128-token sliding window.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 29.9 (Knowledge Work 45.9, Coding 28.3, Science 17, General 28)
- AA Intelligence Index v4.3.2: total 46.3 (General 42.3, Scientific 38, Coding 47.9, Agents 54.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mimo-2.6-pro.webp
- config.json: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/config.json
- Tech report: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/resolve/main/MiMo_V2_6_technical_report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-xiaomi-mimo-v2-6-pro-rl-1-02t
- Web page: https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t/
- JSON: https://llm-atlas-7or.pages.dev/api/models/xiaomi-mimo-v2-6-pro-rl-1-02t.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Step-5 Preview (600B)

> StepFun's 92-layer MoE pairs 512-token sliding-window attention with sparse GQA for a 1M-token context window; the figure shows the text decoder.

| Property | Value |
| --- | --- |
| Lab | StepFun (China) |
| Released | 2026-09-20 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 600B total, 27B active (4.5% active) |
| Context window | 1,048,576 tokens |
| Attention | Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention |
| Layer mix | 69 sliding-window + 23 sparse GQA + 3 MTP layers |
| License | N/A |

**Key detail:** Uses top-512 sparse indexing with block size 8, 352 routed experts with 8 active plus one shared expert, and MTP-3. The first three and final backbone layers use dense FFNs with intermediate size 13,824.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 28.5 (Knowledge Work 40.9, Coding 20.5, Science 14.4, General 38)
- AA Intelligence Index v4.3.2: total 43.7 (General 43, Scientific 33.7, Coding 46.1, Agents 49.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/step-5.webp
- config.json (mirror): https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/config.json
- Tech report: https://www.stepfun.com/step-5-preview
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-step-5-preview-600b
- Web page: https://llm-atlas-7or.pages.dev/models/step-5-preview-600b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/step-5-preview-600b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: DeepSeek V4.1-Flash (552B)

> Causal encoder-decoder MoE with CSA2 cache sharing, Engram memory, and a million-token context. The figure shows the text backbone.

| Property | Value |
| --- | --- |
| Lab | DeepSeek (China) |
| Released | 2026-09-10 |
| Decoder type | Sparse MoE (causal encoder-decoder) |
| Family | Sparse MoE |
| Scale | 552B backbone + 196B Engram; 8B active prefill, 16B active decode (2.9% of backbone) |
| Context window | 1,048,576 tokens |
| Attention | MLA-style CSA2 + SWA with cross-layer KV and sparse-index reuse |
| Layer mix | 20 encoder + 20 decoder layers; 2 SWA-only + 38 CSA2/SWA |
| KV cache / token (bf16) | 3.125 KiB (Very low) |
| License | MIT License |

**Key detail:** Decoder global KV comes from final encoder states. Single-Pass mHC mixes four residual streams around each attention and MoE sublayer.

## Architecture concepts

- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Cross-Layer KV Sharing (KV sharing)](https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md): Later layers reuse key/value states produced by earlier layers.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 39.5 (General 34.3, Scientific 26.8, Coding 39.3, Agents 52.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/deepseek-v4-1-flash.webp
- config.json: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/config.json
- License: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/LICENSE
- Tech report: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-deepseek-v4-1-flash
- Web page: https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash/
- JSON: https://llm-atlas-7or.pages.dev/api/models/deepseek-v4-1-flash.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Tencent Hy4-preview (770B-A49B)

> Tencent's 770B sparse MoE combines gated DeepSeek Sparse Attention, cross-layer index reuse, and four-stream identity Hyper-Connections.

| Property | Value |
| --- | --- |
| Lab | Tencent (China) |
| Released | 2026-08-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 770B backbone, 49B active (6.4% active); plus 10B MTP |
| Context window | 1,048,576 tokens |
| Attention | Gated MLA with DeepSeek Sparse Attention and IndexCache |
| Layer mix | 78 gated MLA/DSA + 1 MTP layer |
| KV cache / token (bf16) | 87.8 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Uses 256 routed experts with 8 routed plus 1 shared expert active per token, a top-2,048 DSA indexer, cross-layer index reuse, and four iHC residual streams.

## Architecture concepts

- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [IndexShare (IndexShare)](https://llm-atlas-7or.pages.dev/concepts/indexshare.md): Nearby layers reuse the token selection computed by one DSA indexer.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/tencent-hy4.webp
- config.json: https://huggingface.co/tencent/Hy4-preview/blob/main/config.json
- License: https://huggingface.co/tencent/Hy4-preview/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-tencent-hy4-preview-770b-a49b
- Web page: https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/tencent-hy4-preview-770b-a49b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-5.3-Flash (320B)

> Natively multimodal GLM refresh, previously previewed as Ox Alpha, that replaces most MLA/DSA layers with Kimi Delta Attention and adds four-stream mHC.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2026-08-26 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 320B total, 18B active (5.6% active) |
| Context window | 1,048,576 tokens |
| Attention | 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention |
| Layer mix | 34 KDA + 11 MLA/DSA + 1 MTP layer |
| KV cache / token (bf16) | 11 KiB (Very low) |
| License | MIT License |

**Key detail:** Mixes 34 Kimi Delta Attention layers with 11 MLA/DSA layers and routes residuals through four mHC streams.

## Architecture concepts

- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 27.8 (Knowledge Work 43.7, Coding 23.7, Science 12.5, General 30.9)
- AA Intelligence Index v4.3.2: total 41.8 (General 39.7, Scientific 27.6, Coding 42.2, Agents 53)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-5.3-flash.webp
- config.json: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/config.json
- License: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2602.15763
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-5-3-flash
- Web page: https://llm-atlas-7or.pages.dev/models/glm-5-3-flash/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-5-3-flash.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3.8-Flash-Next (125B-A6B)

> Experimental vision-language MoE whose text backbone combines Gated DeltaNet, micro-block QSA, four-branch gated residuals, and n-gram embeddings; the figure omits the vision encoder and MTP head.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2026-08-26 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 125B core, 6B active (4.8% active); plus 51B n-gram embeddings and 4B MTP |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA) |
| Layer mix | 12 QSA + 36 DeltaNet |
| KV cache / token (bf16) | 24 KiB (Very low) |
| License | Qwen Community License 1.0 |

**Key detail:** Replaces full attention with micro-block QSA while using four gated residual branches and a 20M-entry n-gram embedding at layer 2.

## Architecture concepts

- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Qwen Sparse Attention (QSA)](https://llm-atlas-7or.pages.dev/concepts/qwen-sparse-attention.md): Like DSA, but the indexer picks whole four-token blocks rather than single tokens.
- [Gated Residuals (GR)](https://llm-atlas-7or.pages.dev/concepts/gated-residuals.md): Gated read/write modules over four residual streams, without mHC's mixing matrix.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Per-Layer Embeddings (PLE)](https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings.md): Each layer gets its own token-specific embedding signal from a large lookup table.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 39.8 (General 35.8, Scientific 24.6, Coding 37.9, Agents 55.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-8-flash-next.webp
- config.json: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
- Tech report: https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-8-flash-next-125b-a6b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-8-flash-next-125b-a6b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ornith 1.5 (35B-A3B)

> Uses the same sparse-hybrid decoder geometry as Qwen3.6 35B-A3B, paired with Ornith's self-improvement training recipe.

| Property | Value |
| --- | --- |
| Lab | Ornith (United States) |
| Released | 2026-08-23 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 35B total, 3B active (8.6% active) |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 10 gated attention + 30 DeltaNet |
| KV cache / token (bf16) | 20 KiB (Very low) |
| License | MIT License |

**Key detail:** Matches Qwen3.6 35B-A3B's 40-layer decoder geometry, with 256 routed experts, top-8 routing, and one shared expert.

## Architecture concepts

- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ornith-1-5-35b-a3b.webp
- config.json: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B/blob/main/config.json
- Tech report: https://ornith.ai/ornith_1_5.html
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ornith-1-5-35b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ornith-1-5-35b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3.8 (27B)

> Dense 27B native vision-language model with a 64-layer Gated DeltaNet and Gated Attention language backbone; the figure omits the vision encoder and MTP head.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2026-08-14 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 27B parameters |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 16 gated attention + 48 DeltaNet |
| KV cache / token (bf16) | 64 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Uses a 64-layer 3:1 Gated DeltaNet/Gated Attention decoder with partial RoPE and one MTP layer.

## Architecture concepts

- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 20.1 (Knowledge Work 39.7, Coding 11.2, Science 1.9, General 27.3)
- AA Intelligence Index v4.3.2: total 33.7 (General 35.9, Scientific 19.7, Coding 26.1, Agents 45.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-8-27b.webp
- config.json: https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-8-27b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-8-27b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-8-27b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nemotron 3.5 Lightning (30B-A3B)

> NVIDIA's 30B hybrid model keeps 3B parameters active and extends the Nemotron 3 Nano backbone with a 1M-token context window and multi-token prediction.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-08-11 |
| Decoder type | Hybrid MoE |
| Family | Hybrid MoE |
| Scale | 30B total, 3B active (10% active) |
| Context window | 1,048,576 tokens |
| Attention | Mostly Mamba-2 with six GQA layers |
| Layer mix | 6 GQA + 23 Mamba-2 + 23 MoE; GQA + MoE MTP predictor |
| KV cache / token (bf16) | 6 KiB (Very low) |
| License | OpenMDW 1.1 |

**Key detail:** Adds a shared GQA-plus-MoE MTP predictor to the 52-layer Nemotron 3 Nano backbone.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 5.5 (Knowledge Work 1, Coding 0.2, Science 0.6, General 20.6)
- AA Intelligence Index v4.3.2: total 12.9 (General 26.3, Scientific 5.3, Coding 16.3, Agents 2.6)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nemotron-3-5-lightning-30b-a3b.webp
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4/blob/main/config.json
- License: https://openmdw.ai/license/1-1/
- Tech report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nemotron-3-5-lightning-30b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/nemotron-3-5-lightning-30b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nemotron-3-5-lightning-30b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Muse Glimmer (30B)

> Meta's dense 29.6B multimodal model combines a 52-layer sandwich-norm decoder with gated GQA and a separate perception encoder; the figure focuses on the text backbone.

| Property | Value |
| --- | --- |
| Lab | Meta (United States) |
| Released | 2026-08-10 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 29.6B parameters (including perception encoder) |
| Context window | 131,072+ tokens |
| Attention | Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention |
| Layer mix | 39 sliding-window + 13 global gated GQA |
| KV cache / token (bf16) | 52 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Applies RoPE only in 2,048-token local layers, uses NoPE in global layers, and gates the attention output before projection.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 6.3 (Knowledge Work 3.4, Coding 1.1, Science 2.8, General 18.1)
- AA Intelligence Index v4.3.2: total 17.5 (General 29.2, Scientific 12.3, Coding 22.7, Agents 5.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/muse-glimmer-30b.webp
- config.json: https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/config.json
- License: https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/LICENSE
- Tech report: https://research.meta.ai/static/muse-glimmer-methodology
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-muse-glimmer-30b
- Web page: https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/muse-glimmer-30b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ling 3.0 Flash (124B)

> Hybrid MoE that alternates five Kimi Delta Attention layers with one gated MLA layer.

| Property | Value |
| --- | --- |
| Lab | InclusionAI (China) |
| Released | 2026-08-04 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 124B total, 5.1B active (4.1% active) |
| Context window | 262,144 tokens |
| Attention | 5:1 Kimi Delta Attention and gated MLA |
| Layer mix | 35 Kimi Delta Attention + 7 gated MLA + 1 MTP layer |
| KV cache / token (bf16) | 7.9 KiB (Very low) |
| License | MIT License |

**Key detail:** Uses fine-grained diagonal KDA gating, top-8 routing over 512 experts plus one shared expert, and one MTP layer.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 20.1 (General 29.3, Scientific 12.7, Coding 21, Agents 15.4)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ling-3-0-flash.webp
- config.json: https://huggingface.co/inclusionAI/Ling-3.0-flash/blob/main/config.json
- Tech report: https://huggingface.co/inclusionAI/Ling-3.0-flash
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ling-3-0-flash
- Web page: https://llm-atlas-7or.pages.dev/models/ling-3-0-flash/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ling-3-0-flash.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kimi K3 (2.8T)

> 2.8T sparse hybrid that combines a 3:1 KDA and Gated MLA stack with Block Attention Residuals and Stable LatentMoE. It is also native multimodal, but the figure focuses on the text portion.

| Property | Value |
| --- | --- |
| Lab | Moonshot AI (China) |
| Released | 2026-07-27 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 2.8T total, 104B active (3.7% active) |
| Context window | 1,048,576 tokens |
| Attention | 3:1 Kimi Delta Attention and Gated MLA with NoPE |
| Layer mix | 69 KDA + 24 Gated MLA |
| KV cache / token (bf16) | 27 KiB (Low) |
| License | Kimi K3 License |

**Key detail:** Uses eight Block Attention Residual groups and Stable LatentMoE with 16 of 896 routed experts active per token.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Attention Residuals (AttnRes)](https://llm-atlas-7or.pages.dev/concepts/attention-residuals.md): Learned, attention-style weights over earlier layer outputs replace the plain residual sum.
- [Latent MoE (Latent MoE)](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md): Routed experts operate in a smaller latent space instead of the full hidden width.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 43.6 (General 45.8, Scientific 35.2, Coding 36.1, Agents 51.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kimi-k3.webp
- config.json: https://huggingface.co/moonshotai/Kimi-K3/blob/main/config.json
- License: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
- Tech report: https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kimi-k3
- Web page: https://llm-atlas-7or.pages.dev/models/kimi-k3/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kimi-k3.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Solar Open 2 (250B)

> Upstage's sparse hybrid alternates one gated GQA layer with three Kimi Delta Attention layers and reaches a 1M-token context without positional embeddings.

| Property | Value |
| --- | --- |
| Lab | Upstage (South Korea) |
| Released | 2026-07-22 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 250B total, 15B active (6% active) |
| Context window | 1,048,576 tokens |
| Attention | 3:1 Kimi Delta Attention and gated GQA with NoPE |
| Layer mix | 12 gated GQA + 36 Kimi Delta Attention |
| KV cache / token (bf16) | 48 KiB (Low) |
| License | Upstage Solar License |

**Key detail:** Uses a repeating gated-GQA/KDA hybrid stack, kernel-4 ShortConv in KDA, and top-8 routing over 320 experts plus one shared expert.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 24.7 (Scientific 17.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/solar-open-2.webp
- config.json: https://huggingface.co/upstage/Solar-Open2-250B/blob/main/config.json
- License: https://huggingface.co/upstage/Solar-Open2-250B/blob/main/LICENSE
- Tech report: https://huggingface.co/upstage/Solar-Open2-250B/blob/main/Solar_Open_2_Tech_Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-solar-open-2
- Web page: https://llm-atlas-7or.pages.dev/models/solar-open-2/
- JSON: https://llm-atlas-7or.pages.dev/api/models/solar-open-2.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Antares (1B)

> Cisco Foundation AI's compact terminal agent uses a dense Granite 4.0 backbone and is post-trained with SFT and GRPO for vulnerability localization.

| Property | Value |
| --- | --- |
| Lab | Cisco (United States) |
| Released | 2026-07-21 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 1B parameters |
| Context window | 131,072 tokens |
| Attention | GQA with RoPE |
| Layer mix | 40 GQA |
| KV cache / token (bf16) | 80 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Retains Granite 4.0's MuP scaling multipliers and tied input/output embeddings, then adds SFT and GRPO for terminal-based vulnerability localization.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/antares-1b.webp
- config.json: https://huggingface.co/fdtn-ai/antares-1b/blob/main/config.json
- Tech report: https://cisco-foundation-ai.github.io/antares/technical-report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-antares-1b
- Web page: https://llm-atlas-7or.pages.dev/models/antares-1b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/antares-1b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Laguna S 2.1 (118B)

> Poolside's 118B sparse MoE scales the Laguna XS recipe to 48 layers and 1M context while activating about 8B parameters per token.

| Property | Value |
| --- | --- |
| Lab | Poolside (United States) |
| Released | 2026-07-21 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 118B total, 8B active (6.8% active) |
| Context window | 1,048,576 tokens |
| Attention | Gated GQA with QK-Norm and 3:1 sliding-window/global attention |
| Layer mix | 36 sliding-window + 12 global GQA |
| KV cache / token (bf16) | 192 KiB (High) |
| License | OpenMDW License 1.1 |

**Key detail:** Uses 256 routed experts with top-10 routing plus one shared expert, a dense first layer, and per-layer query-head counts.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Layer-wise attention budgeting (Attention budgeting)](https://llm-atlas-7or.pages.dev/concepts/attention-budgeting.md): Different layers get different attention budgets: head counts, windows or full context.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/laguna-s-2-1.webp
- config.json: https://huggingface.co/poolside/Laguna-S-2.1/blob/main/config.json
- License: https://huggingface.co/poolside/Laguna-S-2.1/blob/main/LICENSE.md
- Tech report: https://poolside.ai/blog/introducing-laguna-s-2-1
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-laguna-s-2-1
- Web page: https://llm-atlas-7or.pages.dev/models/laguna-s-2-1/
- JSON: https://llm-atlas-7or.pages.dev/api/models/laguna-s-2-1.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Motif 3 Beta (314B)

> Large sparse MoE that combines Grouped Differential Latent Attention, per-expert PolyNorm, and modified mHC in a 53-layer decoder.

| Property | Value |
| --- | --- |
| Lab | Motif Technologies (South Korea) |
| Released | 2026-07-21 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 314B total, 13B active (4.1% active) |
| Context window | 262,144 tokens |
| Attention | GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention |
| Layer mix | 39 sliding-window + 14 full GDLA |
| KV cache / token (bf16) | 530 KiB (Very high) |
| License | Motif Technologies Non-Commercial License |

**Key detail:** Routes each token through 8 of 384 experts plus one shared expert, with per-expert PolyNorm and a 3:1 sliding-window/full GDLA pattern.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [PolyNorm (PolyNorm)](https://llm-atlas-7or.pages.dev/concepts/polynorm.md): A learned activation mixing normalised linear, quadratic and cubic terms.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 32.3 (Scientific 23.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/motif-3-beta.webp
- config.json: https://huggingface.co/Motif-Technologies/Motif-3-Beta/blob/main/config.json
- License: https://huggingface.co/Motif-Technologies/Motif-3-Beta/blob/main/README.md#license
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-motif-3-beta
- Web page: https://llm-atlas-7or.pages.dev/models/motif-3-beta/
- JSON: https://llm-atlas-7or.pages.dev/api/models/motif-3-beta.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nanbeige 4.2 (3B)

> Compact agentic model that runs the same 22-layer dense transformer stack twice, resulting in 44 transformer blocks (layers) without duplicating the layer weights.

| Property | Value |
| --- | --- |
| Lab | BOSS Zhipin (China) |
| Released | 2026-07-21 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 4B total, 3B non-embedding |
| Context window | 262,144 tokens |
| Attention | GQA with RoPE |
| Layer mix | 22 shared GQA layers × 2 passes |
| KV cache / token (bf16) | 176 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Reuses the same 22-layer stack for a second pass, giving 44 block applications with 3B non-embedding parameters.

## Architecture concepts

- [Looped Transformer (Looped depth)](https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing.md): The same blocks are applied several times, reusing weights across depth.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nanbeige-4-2-3b.webp
- config.json: https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/config.json
- Tech report: https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/Nanbeige42_report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nanbeige-4-2-3b
- Web page: https://llm-atlas-7or.pages.dev/models/nanbeige-4-2-3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nanbeige-4-2-3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: BTL-3 (27B)

> Agent-oriented Qwen3.6-27B derivative that adds a rank-32 LoRA adapter for coding, structured tool use, and long multi-turn execution.

| Property | Value |
| --- | --- |
| Lab | Bad Theory Labs (Nigeria) |
| Released | 2026-07-20 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 27B base + 934 MB LoRA adapter |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 16 gated attention + 48 DeltaNet |
| KV cache / token (bf16) | 64 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Retains Qwen3.6-27B's 64-layer hybrid stack while adapting its attention, DeltaNet, and FFN projections through rank-32 LoRA.

## Architecture concepts

- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/btl-3-27b.webp
- adapter_config.json: https://huggingface.co/badtheorylabs/BTL-3/blob/main/adapter_config.json
- License: https://huggingface.co/badtheorylabs/BTL-3/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-btl-3-27b
- Web page: https://llm-atlas-7or.pages.dev/models/btl-3-27b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/btl-3-27b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Inkling (975B)

> Thinking Machines Lab's native multimodal model combines a 66-layer sparse MoE decoder with a 5:1 mix of local and global GQA.

| Property | Value |
| --- | --- |
| Lab | Thinking Machines Lab (United States) |
| Released | 2026-07-15 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 975B total, 41B active (4.2% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with QK-Norm and learned relative-position bias in a 5:1 sliding-window/global pattern |
| Layer mix | 55 sliding-window + 11 global GQA |
| KV cache / token (bf16) | 484 KiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Uses learned relative-position bias instead of RoPE and adds four short convolutions to each decoder layer.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 12.5 (Knowledge Work 13.5, Coding 2.8, Science 7.2, General 26.5)
- AA Intelligence Index v4.3.2: total 25 (General 36.4, Scientific 18.7, Coding 24, Agents 18.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/inkling.webp
- config.json: https://huggingface.co/thinkingmachines/Inkling/blob/main/config.json
- License: https://www.apache.org/licenses/LICENSE-2.0
- Tech report: https://thinkingmachines.ai/news/introducing-inkling/
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-inkling
- Web page: https://llm-atlas-7or.pages.dev/models/inkling/
- JSON: https://llm-atlas-7or.pages.dev/api/models/inkling.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Soofi S (30B-A3B)

> German-English base model that retrains the Nemotron 3 Nano architecture on 26.68T tokens with a fully documented data mixture.

| Property | Value |
| --- | --- |
| Lab | SOOFI Consortium (Germany) |
| Released | 2026-07-10 |
| Decoder type | Hybrid MoE |
| Family | Hybrid MoE |
| Scale | 31.6B total, 3.2B active (10.1% active) |
| Context window | 1,048,576 tokens |
| Attention | Mostly Mamba-2 with 6 GQA layers and no positional embeddings (NoPE) |
| Layer mix | 6 GQA + 23 Mamba-2 + 23 MoE |
| KV cache / token (bf16) | 6 KiB (Very low) |
| License | Custom license (preview) |

**Key detail:** Reuses the Nemotron 3 Nano architecture without modification, isolating the effect of a German-English data and training recipe.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/soofi.webp
- config.json: https://huggingface.co/Soofi-Project/Soofi-S-Base/blob/main/config.json
- License: https://huggingface.co/Soofi-Project/Soofi-S-Base#license-terms-of-use
- Tech report: https://arxiv.org/abs/2607.09424
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-soofi-s-30b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/soofi-s-30b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/soofi-s-30b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Laguna XS 2.1 (33B)

> Poolside's 33B Laguna refresh extends the sparse MoE architecture to 262k context while keeping about 3B parameters active per token.

| Property | Value |
| --- | --- |
| Lab | Poolside (United States) |
| Released | 2026-07-02 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 33B total, 3B active (9.1% active) |
| Context window | 262,144 tokens |
| Attention | Gated GQA with QK-Norm and 3:1 sliding-window/global attention |
| Layer mix | 30 sliding-window + 10 global GQA |
| KV cache / token (bf16) | 160 KiB (Moderate) |
| License | OpenMDW License 1.1 |

**Key detail:** Uses 256 routed experts with top-8 routing plus one shared expert, a dense first layer, and per-layer query-head counts.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Layer-wise attention budgeting (Attention budgeting)](https://llm-atlas-7or.pages.dev/concepts/attention-budgeting.md): Different layers get different attention budgets: head counts, windows or full context.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/laguna-xs-2-1.webp
- config.json: https://huggingface.co/poolside/Laguna-XS-2.1/blob/main/config.json
- License: https://huggingface.co/poolside/Laguna-XS-2.1/blob/main/LICENSE.md
- Tech report: https://huggingface.co/poolside/Laguna-XS-2.1
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-laguna-xs-2-1
- Web page: https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1/
- JSON: https://llm-atlas-7or.pages.dev/api/models/laguna-xs-2-1.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-5.2 (744B)

> Long-context GLM refresh that keeps the 744B sparse MoE backbone and adds IndexShare for 1M-token DSA inference.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2026-06-17 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 744B total, 40B active (5.4% active) |
| Context window | 1,048,576 tokens |
| Attention | MLA with DeepSeek Sparse Attention and IndexShare |
| Layer mix | 78 MLA/DSA |
| KV cache / token (bf16) | 87.8 KiB (Moderate) |
| License | MIT License |

**Key detail:** Reuses each full DSA indexer result across the next three layers, making 1M-token sparse attention cheaper.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [IndexShare (IndexShare)](https://llm-atlas-7or.pages.dev/concepts/indexshare.md): Nearby layers reuse the token selection computed by one DSA indexer.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 33.7 (General 36.9, Scientific 31, Coding 26.1, Agents 37.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-5.2.webp
- config.json: https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2602.15763
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-5-2
- Web page: https://llm-atlas-7or.pages.dev/models/glm-5-2/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-5-2.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: VibeThinker-3B (3B)

> Qwen2.5-Coder-based dense reasoner whose gains mainly come from verifier-driven post-training rather than a new block design.

| Property | Value |
| --- | --- |
| Lab | Weibo (China) |
| Released | 2026-06-15 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 3B parameters |
| Context window | 131,072 tokens |
| Attention | 8:1 GQA with RoPE |
| Layer mix | 36 GQA |
| KV cache / token (bf16) | 36 KiB (Low) |
| License | MIT License |

**Key detail:** Standard Qwen2.5-Coder-style dense stack strengthened by curriculum SFT, MGPO RL, Long2Short RL, offline self-distillation, and Instruct RL.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/vibethinker-3b.webp
- config.json: https://huggingface.co/WeiboAI/VibeThinker-3B/blob/main/config.json
- License: https://huggingface.co/WeiboAI/VibeThinker-3B/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2606.16140
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-vibethinker-3b
- Web page: https://llm-atlas-7or.pages.dev/models/vibethinker-3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/vibethinker-3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: MiniMax M3 (428B)

> Native multimodal MiniMax model that adds sparse attention to an M2-style MoE backbone for 1M-token context.

| Property | Value |
| --- | --- |
| Lab | MiniMax (China) |
| Released | 2026-06-13 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 428B total, 23B active (5.4% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with QK-Norm and MiniMax Sparse Attention |
| Layer mix | 3 full GQA + 57 MiniMax Sparse Attention |
| KV cache / token (bf16) | 120 KiB (Moderate) |
| License | MiniMax Community License |

**Key detail:** Uses three dense prefix layers, then 57 sparse-attention MoE layers with 128 routed experts and 4 routed plus 1 shared expert active per token.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 18.7 (Knowledge Work 23.9, Coding 13.9, Science 10.9, General 26.1)
- AA Intelligence Index v4.3.2: total 29.2 (General 36.3, Scientific 21.3, Coding 24.6, Agents 30.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/minimax-m3.webp
- config.json: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/main/config.json
- License: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2606.13392
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-minimax-m3-428b
- Web page: https://llm-atlas-7or.pages.dev/models/minimax-m3-428b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/minimax-m3-428b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kimi K2.7 Code (1T)

> Coding-focused K2.6 successor that keeps the same 1T sparse MoE text backbone and emphasizes long-horizon software engineering agents.

| Property | Value |
| --- | --- |
| Lab | Moonshot AI (China) |
| Released | 2026-06-12 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1T total, 32B active (3.2% active) |
| Context window | 256,000 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | Modified MIT License |

**Key detail:** Keeps the Kimi K2.5/K2.6 text architecture and shifts the release focus toward coding-agent workflows.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 25.8 (General 33.1, Scientific 22.5, Coding 24.4, Agents 22)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kimi-k2-7-code.webp
- config.json: https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/config.json
- License: https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/LICENSE
- Tech report: https://huggingface.co/moonshotai/Kimi-K2.7-Code
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kimi-k2-7-code
- Web page: https://llm-atlas-7or.pages.dev/models/kimi-k2-7-code/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kimi-k2-7-code.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: North Mini Code (30B-A3B)

> Cohere's compact agentic coding MoE adapts the parallel Cohere2-MoE block to a 30B total, 3B active model.

| Property | Value |
| --- | --- |
| Lab | Cohere (Canada) |
| Released | 2026-06-05 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 30B total, 3B active (10% active) |
| Context window | 256K input, 64K output tokens |
| Attention | 8:1 GQA with 3:1 sliding-window/global attention |
| Layer mix | 36 sliding-window + 13 global GQA/MoE |
| KV cache / token (bf16) | 98 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Runs attention and MoE in parallel, with RoPE in sliding-window layers, NoPE in global layers, and top-8 routing over 128 experts.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9.9 (General 15.9, Scientific 5.7, Coding 19.6, Agents 0.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/north-mini-code.webp
- config.json: https://huggingface.co/CohereLabs/North-Mini-Code-1.0/blob/main/config.json
- Tech report: https://huggingface.co/blog/CohereLabs/introducing-north-mini-code
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-north-mini-code-30b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/north-mini-code-30b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nemotron 3 Ultra (550B-A55B)

> NVIDIA's Ultra model scales the Nemotron 3 hybrid stack to 550B total parameters while keeping 55B active per token.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-06-04 |
| Decoder type | Hybrid MoE |
| Family | Hybrid MoE |
| Scale | 550B total, 55B active (10% active) |
| Context window | 1,000,000 tokens |
| Attention | Mostly Mamba-2 with a few GQA layers |
| Layer mix | 12 GQA + 48 Mamba-2 + 48 MoE |
| KV cache / token (bf16) | 12 KiB (Very low) |
| License | NVIDIA Nemotron Open Model License |

**Key detail:** Scales the latent-MoE hybrid to 108 layers with 48 Mamba-2, 48 latent MoE, and 12 GQA layers.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 11.8 (Knowledge Work 12.9, Coding 0.9, Science 6, General 27.4)
- AA Intelligence Index v4.3.2: total 22.9 (General 34.1, Scientific 15.8, Coding 20.4, Agents 18.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nemotron-3-ultra-550b-a55b.webp
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4/blob/main/config.json
- License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
- Tech report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nemotron-3-ultra-550b-a55b
- Web page: https://llm-atlas-7or.pages.dev/models/nemotron-3-ultra-550b-a55b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nemotron-3-ultra-550b-a55b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 4 (12B)

> Dense Gemma 4 12B Unified uses an encoder-free multimodal input path, while its text decoder keeps the 31B-style local-global attention recipe.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2026-06-03 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 12B parameters |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention |
| Layer mix | 40 sliding-window + 8 global |
| KV cache / token (bf16) | 328 KiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Figure shows the 48-layer text decoder; the full 12B Unified model also projects audio, image, and video inputs directly into the decoder input stream.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9.4 (Scientific 3.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-4-12b.webp
- config.json: https://huggingface.co/google/gemma-4-12B-it/blob/main/config.json
- Tech report: https://developers.googleblog.com/gemma-4-12b-the-developer-guide/
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-4-12b
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-4-12b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-4-12b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: JetBrains Mellum2 Thinking (12B-A2.5B)

> JetBrains' Mellum2 Thinking is a compact reasoning-tuned sparse MoE for code and agentic workflows.

| Property | Value |
| --- | --- |
| Lab | JetBrains (Czech Republic) |
| Released | 2026-06-01 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 12B total, 2.5B active (20.8% active) |
| Context window | 131,072 tokens |
| Attention | GQA with 3:1 sliding-window/full attention |
| Layer mix | 21 sliding-window + 7 full GQA |
| KV cache / token (bf16) | 56 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Uses 64 experts with 8 active per token, a 1024-token sliding window, and YaRN scaling on full-attention layers.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mellum2.webp
- config.json: https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking/blob/main/config.json
- Tech report: https://arxiv.org/abs/2605.31268
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-jetbrains-mellum2-thinking-12b-a2-5b
- Web page: https://llm-atlas-7or.pages.dev/models/jetbrains-mellum2-thinking-12b-a2-5b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/jetbrains-mellum2-thinking-12b-a2-5b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: LFM2.5 (1.2B)

> Liquid AI's 1.2B LFM2.5 model keeps the 16-layer hybrid LIV convolution and GQA stack at a wider hidden size.

| Property | Value |
| --- | --- |
| Lab | Liquid AI (United States) |
| Released | 2026-05-28 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 1.2B parameters |
| Context window | 128,000 tokens |
| Attention | LIV convolution blocks plus GQA |
| Layer mix | 10 LIV conv + 6 GQA |
| KV cache / token (bf16) | 12 KiB (Very low) |
| License | LFM Open License v1.0 |

**Key detail:** Keeps the 16-layer LFM2.5 hybrid stack but doubles the hidden size to 2048 and uses 32 query heads.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.2 (Scientific 3.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/lfm2-5-1-2b.webp
- config.json: https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/config.json
- License: https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/LICENSE
- Tech report: https://www.liquid.ai/blog/lfm2-5-1-2b-thinking-on-device-reasoning-under-1gb
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-lfm2-5-1-2b
- Web page: https://llm-atlas-7or.pages.dev/models/lfm2-5-1-2b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/lfm2-5-1-2b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: LFM2.5 (350M)

> Liquid AI's compact LFM2.5 model combines double-gated LIV convolution blocks with periodic GQA layers.

| Property | Value |
| --- | --- |
| Lab | Liquid AI (United States) |
| Released | 2026-05-28 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 350M parameters |
| Context window | 128,000 tokens |
| Attention | LIV convolution blocks plus GQA |
| Layer mix | 10 LIV conv + 6 GQA |
| KV cache / token (bf16) | 12 KiB (Very low) |
| License | LFM Open License v1.0 |

**Key detail:** Uses a 16-layer hybrid stack with 10 LIV convolution layers and 6 GQA layers.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/lfm2-5-350m.webp
- config.json: https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/config.json
- License: https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/LICENSE
- Tech report: https://www.liquid.ai/blog/lfm2-5-350m-no-size-left-behind
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-lfm2-5-350m
- Web page: https://llm-atlas-7or.pages.dev/models/lfm2-5-350m/
- JSON: https://llm-atlas-7or.pages.dev/api/models/lfm2-5-350m.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: LFM2.5 (8B-A1B)

> Liquid AI's sparse LFM2.5 model combines LIV convolution, GQA, and MoE routing with 4 of 32 experts active per token.

| Property | Value |
| --- | --- |
| Lab | Liquid AI (United States) |
| Released | 2026-05-28 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 8.3B total, 1.5B active (18.1% active) |
| Context window | 128,000 tokens |
| Attention | LIV convolution blocks plus GQA and MoE |
| Layer mix | 18 LIV conv + 6 GQA + MoE FFNs |
| KV cache / token (bf16) | 12 KiB (Very low) |
| License | LFM Open License v1.0 |

**Key detail:** Uses 24 layers, 32 MoE experts with 4 active per token, and leaves the first two layers dense before sparse routing begins.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 7.2 (Scientific 3.4)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/lfm2-5-8b-a1b.webp
- config.json: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/config.json
- License: https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE
- Tech report: https://www.liquid.ai/blog/lfm2-5-8b-a1b
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-lfm2-5-8b-a1b
- Web page: https://llm-atlas-7or.pages.dev/models/lfm2-5-8b-a1b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/lfm2-5-8b-a1b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Command A+ (218B-A25B)

> Cohere's open sparse MoE model scales the Command A parallel transformer block to 218B total parameters while keeping 25B active per token.

| Property | Value |
| --- | --- |
| Lab | Cohere (Canada) |
| Released | 2026-05-20 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 218B total, 25B active/token (11.5% active) |
| Context window | 128K input, 64K output tokens |
| Attention | 16:1 GQA with 3:1 sliding-window/global attention |
| Layer mix | 24 sliding-window + 8 global GQA/MoE |
| KV cache / token (bf16) | 128 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Runs attention and MoE in parallel, which Cohere reports gives similar quality to a sequential block with better throughput.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 6.2 (Knowledge Work 0.6, Coding 0.2, Science 1, General 23.2)
- AA Intelligence Index v4.3.2: total 13.1 (General 26.4, Scientific 6.1, Coding 19.5, Agents 0.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/command-a-plus.webp
- config.json: https://huggingface.co/CohereLabs/command-a-plus-05-2026-w4a4/blob/main/config.json
- Tech report: https://cohere.com/blog/command-a-plus
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-command-a-218b-a25b
- Web page: https://llm-atlas-7or.pages.dev/models/command-a-218b-a25b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/command-a-218b-a25b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: ZAYA1-8B (8.4B)

> Zyphra's compact reasoning MoE pairs CCA/GQA attention updates with top-1 routed expert FFN updates to keep fewer than 1B parameters active per token.

| Property | Value |
| --- | --- |
| Lab | Zyphra (United States) |
| Released | 2026-05-06 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 8.4B total, 760M active/token (9% active) |
| Context window | 131,072 tokens |
| Attention | CCA with 4:1 GQA, RoPE, and Q/K L2 norm |
| Layer mix | 40 CCA/GQA attention + 40 top-1 MoE FFN entries |
| KV cache / token (bf16) | 40 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Key detail is CCA/GQA attention, which performs attention in a compressed latent space while keeping a 4:1 GQA layout. Sidenote: the config lists 80 alternating layer entries, equivalent to 40 CCA/GQA attention updates paired with 40 top-1 MoE FFN updates.

## Architecture concepts

- [Compressed Convolutional Attention (CCA)](https://llm-atlas-7or.pages.dev/concepts/compressed-convolutional-attention.md): Attention computed directly in a compressed latent space, with convolutional mixing.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/zaya1-8b.webp
- View in article: https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures#%C2%A74-compressed-convolutional-attention-zaya1-8b
- config.json: https://huggingface.co/Zyphra/ZAYA1-8B/blob/main/config.json
- Tech report: https://arxiv.org/abs/2605.05365
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-zaya1-8b
- Web page: https://llm-atlas-7or.pages.dev/models/zaya1-8b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/zaya1-8b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Granite 4.1 (30B)

> IBM's dense 30B Granite 4.1 model keeps a straightforward long-context GQA transformer stack and improves tool calling, instruction following, and chat behavior through post-training.

| Property | Value |
| --- | --- |
| Lab | IBM (United States) |
| Released | 2026-04-29 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 30B parameters |
| Context window | 131,072 tokens |
| Attention | GQA with RoPE |
| Layer mix | 64 GQA |
| KV cache / token (bf16) | 256 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Uses a 4096-wide dense stack, 32 query heads, 8 KV heads, SwiGLU MLPs, shared input/output embeddings, and Granite-specific MuP scaling factors.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 7.4 (Scientific 2.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/granite-4.1.webp
- config.json: https://huggingface.co/ibm-granite/granite-4.1-30b/blob/main/config.json
- Tech report: https://huggingface.co/blog/ibm-granite/granite-4-1
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-granite-4-1-30b
- Web page: https://llm-atlas-7or.pages.dev/models/granite-4-1-30b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/granite-4-1-30b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Laguna XS.2 (33B)

> Poolside's open-weight agentic coding MoE uses mixed sliding-window/global attention and activates only 3B of 33B parameters per token.

| Property | Value |
| --- | --- |
| Lab | Poolside (United States) |
| Released | 2026-04-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 33B total, 3B active (9.1% active) |
| Context window | 131,072 tokens |
| Attention | Gated GQA with QK-Norm and 3:1 sliding-window/global attention |
| Layer mix | 30 sliding-window + 10 global |
| KV cache / token (bf16) | 160 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Uses per-layer query-head counts, a 512-token SWA window, sigmoid MoE routing, and one shared expert alongside the top-8 routed experts.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Layer-wise attention budgeting (Attention budgeting)](https://llm-atlas-7or.pages.dev/concepts/attention-budgeting.md): Different layers get different attention budgets: head counts, windows or full context.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/laguna-xs2.webp
- View in article: https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures#%C2%A73-layer-wise-attention-budgeting-laguna-xs2
- config.json: https://huggingface.co/poolside/Laguna-XS.2/blob/main/config.json
- License: https://huggingface.co/poolside/Laguna-XS.2/blob/main/LICENSE.md
- Tech report: https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-laguna-xs-2
- Web page: https://llm-atlas-7or.pages.dev/models/laguna-xs-2/
- JSON: https://llm-atlas-7or.pages.dev/api/models/laguna-xs-2.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: DeepSeek V4-Flash (284B)

> DeepSeek's efficient V4 preview keeps the million-token architecture while reducing the MoE scale to 284B parameters and 13B active parameters.

| Property | Value |
| --- | --- |
| Lab | DeepSeek (China) |
| Released | 2026-04-24 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 284B total, 13B active (4.6% active) |
| Context window | 1,048,576 tokens |
| Attention | MLA-style CSA/HCA with mHC |
| Layer mix | 43 CSA/HCA |
| KV cache / token (bf16) | 5.4 KiB (Very low) |
| License | MIT License |

**Key detail:** Uses 256 experts, 6 routed plus 1 shared expert per token, hash-based routing, compressed attention, and the V4 MTP path.

## Architecture concepts

- [Compressed Sparse & Heavily Compressed Attention (CSA / HCA)](https://llm-atlas-7or.pages.dev/concepts/csa-hca.md): Pool groups of older tokens into fewer KV entries, compressing along the sequence axis.
- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 34.3 (General 31.8, Scientific 27.6, Coding 31.2, Agents 43.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/deepseek-v4-flash.webp
- View in article: https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures#%C2%A75-csahca-mhc-and-compressed-attention-caches-deepseek-v4
- config.json: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/config.json
- License: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2606.19348
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-deepseek-v4-flash
- Web page: https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash/
- JSON: https://llm-atlas-7or.pages.dev/api/models/deepseek-v4-flash.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: DeepSeek V4-Pro (1.6T)

> DeepSeek's flagship V4 preview scales to 1.6T parameters and introduces compressed sparse attention plus manifold-constrained hyper-connections for million-token contexts.

| Property | Value |
| --- | --- |
| Lab | DeepSeek (China) |
| Released | 2026-04-24 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1.6T total, 49B active (3.1% active) |
| Context window | 1,048,576 tokens |
| Attention | MLA-style CSA/HCA with mHC |
| Layer mix | 61 CSA/HCA |
| KV cache / token (bf16) | 7.7 KiB (Very low) |
| License | MIT License |

**Key detail:** Uses 384 experts, 6 routed plus 1 shared expert per token, hash-based routing, compressed attention caches, and the V4 MTP path.

## Architecture concepts

- [Compressed Sparse & Heavily Compressed Attention (CSA / HCA)](https://llm-atlas-7or.pages.dev/concepts/csa-hca.md): Pool groups of older tokens into fewer KV entries, compressing along the sequence axis.
- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 23.1 (Knowledge Work 36.3, Coding 15.9, Science 12.8, General 27.1)
- AA Intelligence Index v4.3.2: total 36 (General 34.4, Scientific 29.5, Coding 32.6, Agents 44.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/deepseek-v4-pro.webp
- View in article: https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures#%C2%A75-csahca-mhc-and-compressed-attention-caches-deepseek-v4
- config.json: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/config.json
- License: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2606.19348
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-deepseek-v4-pro
- Web page: https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro/
- JSON: https://llm-atlas-7or.pages.dev/api/models/deepseek-v4-pro.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ling 2.6 (1T)

> Ling 2.5 successor that keeps the Lightning Attention and MLA hybrid stack while adding an MTP layer for multi-token prediction.

| Property | Value |
| --- | --- |
| Lab | InclusionAI (China) |
| Released | 2026-04-23 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 1T total, 63B active (6.3% active) |
| Context window | 262,144 tokens |
| Attention | Lightning Attention plus MLA |
| Layer mix | 10 MLA + 70 Lightning Attention + 1 MTP layer |
| KV cache / token (bf16) | 11.2 KiB (Very low) |
| License | MIT License |

**Key detail:** Keeps the 7:1 linear-attention/MLA ratio and adds one multi-token-prediction layer for speculative decoding.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 17 (Scientific 4.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ling-2-6-1t.webp
- config.json: https://huggingface.co/inclusionAI/Ling-2.6-1T/blob/main/config.json
- Tech report: https://huggingface.co/inclusionAI/Ling-2.6-1T
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ling-2-6-1t
- Web page: https://llm-atlas-7or.pages.dev/models/ling-2-6-1t/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ling-2-6-1t.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Tencent Hy3-preview (295B-A21B)

> Tencent Hunyuan's 295B sparse MoE preview combines a dense first FFN layer, GQA with QK-Norm, and one extra MTP layer for faster generation.

| Property | Value |
| --- | --- |
| Lab | Tencent (China) |
| Released | 2026-04-23 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 295B total, 21B active (7.1% active) |
| Context window | 262,144 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 80 GQA + 1 MTP layer |
| KV cache / token (bf16) | 320 KiB (Very high) |
| License | Tencent Hy Community License Agreement |

**Key detail:** Uses 192 routed experts, 8 routed plus 1 shared expert active per token, sigmoid routing with expert bias, and a 3.8B MTP layer.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 17 (Scientific 3.6)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/tencent-hy3.webp
- config.json: https://huggingface.co/tencent/Hy3-preview/blob/main/config.json
- License: https://huggingface.co/tencent/Hy3-preview/blob/main/LICENSE
- Tech report: https://www.tencent.com/en-us/articles/2202320.html
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-tencent-hy3-preview-295b-a21b
- Web page: https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/tencent-hy3-preview-295b-a21b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3.6 (27B)

> Dense Qwen3.6 model that keeps the Qwen3.5-style Gated DeltaNet/Gated Attention hybrid stack while replacing MoE blocks with dense FFNs.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2026-04-22 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 27B parameters |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 16 gated attention + 48 DeltaNet |
| KV cache / token (bf16) | 64 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Uses a 64-layer dense hybrid layout with 48 DeltaNet layers and 16 full-attention layers.

## Architecture concepts

- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 21.4 (General 31.5, Scientific 12.1, Coding 21.4, Agents 17.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-6-27b.webp
- config.json: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
- Tech report: https://qwen.ai/blog?id=qwen3.6-27b
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-6-27b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-6-27b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-6-27b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Xiaomi MiMo-V2.5 (310B)

> Omnimodal sparse MoE model that extends the MiMo-V2-Flash backbone with vision and audio encoders.

| Property | Value |
| --- | --- |
| Lab | Xiaomi (China) |
| Released | 2026-04-22 |
| Decoder type | Sparse omnimodal MoE |
| Family | Sparse MoE |
| Scale | 310B total, 15B active (4.8% active) |
| Context window | 1,048,576 tokens |
| Attention | 5:1 sliding-window/global attention |
| Layer mix | 39 sliding-window + 9 global |
| KV cache / token (bf16) | 144 KiB (Moderate) |
| License | MIT License |

**Key detail:** Adds native image, video, and audio encoders to the MiMo-V2-Flash-style sparse MoE backbone with MTP support.

## Architecture concepts

- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 21.8 (General 30.4, Scientific 15.5, Coding 21.9, Agents 17.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/xiaomi-mimo-v2-5-310b.webp
- config.json: https://huggingface.co/XiaomiMiMo/MiMo-V2.5/blob/main/config.json
- Tech report: https://mimo.xiaomi.com/mimo-v2-5
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-xiaomi-mimo-v2-5-310b
- Web page: https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-310b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/xiaomi-mimo-v2-5-310b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Xiaomi MiMo-V2.5-Pro (1.02T)

> Trillion-parameter sparse MoE model for long-horizon agentic and software-engineering tasks.

| Property | Value |
| --- | --- |
| Lab | Xiaomi (China) |
| Released | 2026-04-22 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1.02T total, 42B active (4.1% active) |
| Context window | 1,048,576 tokens |
| Attention | GQA with 6:1 sliding-window/global attention |
| Layer mix | 60 sliding-window + 10 global |
| KV cache / token (bf16) | 350 KiB (Very high) |
| License | MIT License |

**Key detail:** Scales the MiMo-V2.5 Pro stack to 70 layers, 384 routed experts, top-8 routing, and a 1M-token context window.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 26 (General 34.6, Scientific 19.8, Coding 25.3, Agents 22.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/xiaomi-mimo25-pro.webp
- config.json: https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro/blob/main/config.json
- Tech report: https://mimo.xiaomi.com/mimo-v2-5
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-xiaomi-mimo-v2-5-pro-1-02t
- Web page: https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-pro-1-02t/
- JSON: https://llm-atlas-7or.pages.dev/api/models/xiaomi-mimo-v2-5-pro-1-02t.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kimi K2.6 (1T)

> Native-multimodal K2.5 successor that keeps the same 1T sparse MoE backbone while targeting stronger long-horizon coding, design, and agent orchestration.

| Property | Value |
| --- | --- |
| Lab | Moonshot AI (China) |
| Released | 2026-04-20 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1T total, 32B active (3.2% active) |
| Context window | 256,000 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | Modified MIT License |

**Key detail:** Uses the same text architecture as Kimi K2.5, with the main change coming from the multimodal and agentic training recipe.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 27 (General 38.6, Scientific 22.7, Coding 26, Agents 19.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kimi-k2-6.webp
- config.json: https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/config.json
- License: https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/LICENSE
- Tech report: https://www.kimi.com/blog/kimi-k2-6.html
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kimi-k2-6
- Web page: https://llm-atlas-7or.pages.dev/models/kimi-k2-6/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kimi-k2-6.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3.6 (35B-A3B)

> Compact Qwen3.6 open-weight MoE that keeps the Qwen3.5 hybrid Gated DeltaNet/Gated Attention recipe while activating only about 3B parameters.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2026-04-15 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 35B total, 3B active (8.6% active) |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 10 gated attention + 30 DeltaNet |
| KV cache / token (bf16) | 20 KiB (Very low) |
| License | Apache License 2.0 |

**Key detail:** Uses 256 experts with 8 routed plus 1 shared expert active inside a 40-layer hybrid stack.

## Architecture concepts

- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 18.2 (General 28.7, Scientific 11.3, Coding 18.3, Agents 12.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-6-35b-a3b.webp
- config.json: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/LICENSE
- Tech report: https://qwen.ai/blog?id=qwen3.6-35b-a3b
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-6-35b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-6-35b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-6-35b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-5.1 (744B)

> Post-trained GLM refresh that keeps the GLM-5 backbone intact but targets stronger long-horizon agentic coding.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2026-04-07 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 744B total, 40B active (5.4% active) |
| Context window | 202,752 tokens |
| Attention | MLA with DeepSeek Sparse Attention |
| Layer mix | 78 MLA |
| KV cache / token (bf16) | 87.8 KiB (Moderate) |
| License | MIT License |

**Key detail:** Architecture stays aligned with GLM-5, including its MTP-capable backbone; the main shift is post-training.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 26.1 (General 34.7, Scientific 17.3, Coding 23.4, Agents 25.3)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-5-1.webp
- config.json: https://huggingface.co/zai-org/GLM-5.1/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2602.15763
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-5-1
- Web page: https://llm-atlas-7or.pages.dev/models/glm-5-1/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-5-1.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 4 (31B)

> Dense Gemma 4 scales the family to a 256K-context multimodal checkpoint without changing the core local-global recipe much.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2026-04-02 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 30.7B parameters |
| Context window | 256,000 tokens |
| Attention | GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention |
| Layer mix | 50 sliding-window + 10 global |
| KV cache / token (bf16) | 840 KiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Carries Gemma's unusual pre/post-norm stack into a larger 31B dense model with 256K context.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 13.9 (Scientific 5.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-4-31b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A723-gemma-4
- config.json: https://huggingface.co/google/gemma-4-31B-it/blob/main/config.json
- Tech report: https://ai.google.dev/gemma/docs/core/model_card_4
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-4-31b
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-4-31b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-4-31b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 4 (26B-A4B)

> Sparse Gemma 4 variant that keeps the local:global attention backbone while swapping dense FFNs for MoE layers.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2026-04-02 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 25.2B total, 3.8B active (15.1% active) |
| Context window | 256,000 tokens |
| Attention | GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention |
| Layer mix | 25 sliding-window + 5 global |
| KV cache / token (bf16) | 210 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Uses 128 total experts with only 8 routed plus 1 shared expert active per token.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 13.1 (Scientific 5.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-4-26b-a4b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A723-gemma-4
- config.json: https://huggingface.co/google/gemma-4-26B-A4B-it/blob/main/config.json
- Tech report: https://ai.google.dev/gemma/docs/core/model_card_4
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-4-26b-a4b
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-4-26b-a4b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 4 (E2B)

> Smallest Gemma 4 edge model keeps the family's hybrid attention stack and adds native audio on a phone-scale multimodal footprint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 2.3B than a full 5.1B dense model.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2026-04-02 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 5.1B parameters (2.3B effective) |
| Context window | 128,000 tokens |
| Attention | Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention |
| Layer mix | 28 sliding-window + 7 global |
| KV cache / token (bf16) | 35 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Uses a double-wide GELU MLP plus a single KV head to stay light enough for offline edge deployments.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Cross-Layer KV Sharing (KV sharing)](https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md): Later layers reuse key/value states produced by earlier layers.
- [Per-Layer Embeddings (PLE)](https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings.md): Each layer gets its own token-specific embedding signal from a large lookup table.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6.5 (Scientific 2.4)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-4-e2b.webp
- View in article: https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures#%C2%A71-reusing-kv-tensors-across-layers-to-shrink-the-cache-gemma-4
- From scratch: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/17_gemma4/standalone-gemma4-plus-kvcache.ipynb
- config.json: https://huggingface.co/google/gemma-4-E2B-it/blob/main/config.json
- Tech report: https://ai.google.dev/gemma/docs/core/model_card_4
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-4-e2b
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-4-e2b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-4-e2b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 4 (E4B)

> Larger Gemma 4 edge variant keeps the same multimodal hybrid recipe but doubles width and KV heads for a stronger 128K mobile checkpoint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 4.5B than a full 8B dense model.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2026-04-02 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 8B parameters (4.5B effective) |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention |
| Layer mix | 35 sliding-window + 7 global |
| KV cache / token (bf16) | 84 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Steps up to a 42-layer stack with 2 KV heads while keeping the same edge-oriented local/global template.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Cross-Layer KV Sharing (KV sharing)](https://llm-atlas-7or.pages.dev/concepts/kv-sharing.md): Later layers reuse key/value states produced by earlier layers.
- [Per-Layer Embeddings (PLE)](https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings.md): Each layer gets its own token-specific embedding signal from a large lookup table.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 7.5 (Scientific 2.6)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-4-e4b.webp
- View in article: https://magazine.sebastianraschka.com/p/recent-developments-in-llm-architectures#%C2%A71-reusing-kv-tensors-across-layers-to-shrink-the-cache-gemma-4
- From scratch: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/17_gemma4/standalone-gemma4-plus-kvcache.ipynb
- config.json: https://huggingface.co/google/gemma-4-E4B-it/blob/main/config.json
- Tech report: https://ai.google.dev/gemma/docs/core/model_card_4
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-4-e4b
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-4-e4b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-4-e4b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: MiniMax M2.7 (230B)

> Agent-focused 230B follow-up built for self-evolution, agent harnesses, and productivity workflows.

| Property | Value |
| --- | --- |
| Lab | MiniMax (China) |
| Released | 2026-03-18 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 230B total, 10B active (4.3% active) |
| Context window | 196,608 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 62 GQA |
| KV cache / token (bf16) | 248 KiB (High) |
| License | MiniMax M2.7 Non-Commercial License |

**Key detail:** Keeps the M2.5-style 62-layer sparse MoE stack and adds three MTP modules for speculative decoding support.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 22.8 (General 34.7, Scientific 15.1, Coding 25.1, Agents 14.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/minimax-m2-7-230b.webp
- config.json: https://huggingface.co/MiniMaxAI/MiniMax-M2.7/blob/main/config.json
- License: https://huggingface.co/MiniMaxAI/MiniMax-M2.7/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2605.26494
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-minimax-m2-7-230b
- Web page: https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/minimax-m2-7-230b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Mistral Small 4 (119B)

> Multimodal Mistral Small refresh that jumps from the older dense 24B stack to an MLA-based sparse MoE design.

| Property | Value |
| --- | --- |
| Lab | Mistral AI (France) |
| Released | 2026-03-16 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 119B total, 6.63B active (5.6% active) |
| Context window | 256,000 tokens |
| Attention | MLA |
| Layer mix | 36 MLA |
| KV cache / token (bf16) | 22.5 KiB (Very low) |
| License | Apache License 2.0 |

**Key detail:** Uses 128 experts with 4 routed plus 1 shared expert active per token while unifying instruct, reasoning, and vision.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 4.3 (Knowledge Work 0.2, Coding 0, Science 0, General 17)
- AA Intelligence Index v4.3.2: total 11.3 (General 21.1, Scientific 5.1, Coding 19.4, Agents 0.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mistral-small-4.webp
- config.json: https://huggingface.co/mistralai/Mistral-Small-4-119B-2603/blob/main/config.json
- Tech report: https://mistral.ai/news/mistral-small-4
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-mistral-small-4
- Web page: https://llm-atlas-7or.pages.dev/models/mistral-small-4/
- JSON: https://llm-atlas-7or.pages.dev/api/models/mistral-small-4.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nemotron 3 Nano (4B)

> Compact on-device hybrid that compresses Nemotron Nano 9B v2 into a mostly Mamba-2 stack with only four attention layers.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-03-16 |
| Decoder type | Dense hybrid |
| Family | Dense hybrid |
| Scale | 4B parameters |
| Context window | 262,144 tokens |
| Attention | GQA with only 4 attention layers |
| Layer mix | 4 GQA + 21 Mamba-2 + 17 FFN |
| KV cache / token (bf16) | 16 KiB (Very low) |
| License | NVIDIA Nemotron Open Model License |

**Key detail:** Uses a 42-layer stack with 21 Mamba-2 blocks, 17 ReLU² FFNs, and just 4 GQA layers.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v5.0: total 1.8 (Knowledge Work 0.2, Coding 0, Science 0, General 7.1)
- AA Intelligence Index v4.3.2: total 4.8 (General 8.8, Scientific 2.4, Coding 8, Agents 0.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nemotron-3-nano-4b.webp
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16/blob/main/config.json
- License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
- Tech report: https://huggingface.co/blog/nvidia/nemotron-3-nano-4b
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nemotron-3-nano-4b
- Web page: https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-4b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nemotron-3-nano-4b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nemotron 3 Super (120B-A12B)

> The Super variant scales up Nano and adds both latent experts and native speculative decoding support.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-03-11 |
| Decoder type | Hybrid MoE |
| Family | Hybrid MoE |
| Scale | 120B total, 12B active (10% active) |
| Context window | 1,000,000 tokens |
| Attention | Mostly Mamba-2 with a few GQA layers |
| Layer mix | 8 GQA + 40 Mamba-2 + 40 MoE |
| KV cache / token (bf16) | 8 KiB (Very low) |
| License | NVIDIA Nemotron Open Model License |

**Key detail:** Adds latent-space MoE and shared-weight MTP for fast inference.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Latent MoE (Latent MoE)](https://llm-atlas-7or.pages.dev/concepts/latent-moe.md): Routed experts operate in a smaller latent space instead of the full hidden width.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 12.8 (General 22.1, Scientific 12, Coding 18.1, Agents 0.6)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nemotron-3-super-120b-a12b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A7182-nemotron-3-super
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16/blob/main/config.json
- License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
- Tech report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Super-Technical-Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nemotron-3-super-120b-a12b
- Web page: https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nemotron-3-super-120b-a12b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Sarvam (30B)

> Reasoning-oriented Indian-language sparse MoE that keeps GQA at the smaller size.

| Property | Value |
| --- | --- |
| Lab | Sarvam AI (India) |
| Released | 2026-03-03 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 30B total, 2.4B active (8% active) |
| Context window | 131,072 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 19 GQA |
| KV cache / token (bf16) | 19 KiB (Very low) |
| License | Apache License 2.0 |

**Key detail:** Large vocabulary and strong Indic language support paired with a reasoning-focused sparse MoE design.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6.6 (Scientific 3.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/sarvam-30b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A7update-1-sarvam-30b-and-105b-mar-6-2026
- config.json: https://huggingface.co/sarvamai/sarvam-30b/blob/main/config.json
- Tech report: https://www.sarvam.ai/blogs/sarvam-30b-105b
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-sarvam-30b
- Web page: https://llm-atlas-7or.pages.dev/models/sarvam-30b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/sarvam-30b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Sarvam (105B)

> Larger Sarvam variant keeps the sparse MoE layout but switches from GQA to MLA.

| Property | Value |
| --- | --- |
| Lab | Sarvam AI (India) |
| Released | 2026-03-03 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 105B total, 10.3B active (9.8% active) |
| Context window | 131,072 tokens |
| Attention | MLA with KV LayerNorm and NoPE + RoPE |
| Layer mix | 32 MLA |
| KV cache / token (bf16) | 36 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Large vocabulary and strong Indic language support carried into the larger MLA-based sparse MoE variant.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 8.8 (Scientific 5.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/sarvam-105b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A7update-1-sarvam-30b-and-105b-mar-6-2026
- config.json: https://huggingface.co/sarvamai/sarvam-105b/blob/main/config.json
- Tech report: https://www.sarvam.ai/blogs/sarvam-30b-105b
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-sarvam-105b
- Web page: https://llm-atlas-7or.pages.dev/models/sarvam-105b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/sarvam-105b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3.5 (397B)

> Mainline Qwen refresh that brings the Next-style hybrid attention into the flagship series.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2026-02-16 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 397B total, 17B active (4.3% active) |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 15 gated attention + 45 DeltaNet |
| KV cache / token (bf16) | 30 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Turns the former Qwen3-Next side branch into the new core design with 512 experts and 17B active parameters.

## Architecture concepts

- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 21.4 (Scientific 10.3)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-5-397b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A78-qwen35-and-the-continuation-of-hybrid-attention
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/16_qwen3.5
- config.json: https://huggingface.co/Qwen/Qwen3.5-397B-A17B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3.5-397B-A17B/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-5-397b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-5-397b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-5-397b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ling 2.5 (1T)

> Trillion-parameter long-context model that swaps DeltaNet for Lightning Attention.

| Property | Value |
| --- | --- |
| Lab | InclusionAI (China) |
| Released | 2026-02-15 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 1T total, 63B active (6.3% active) |
| Context window | 256,000 tokens |
| Attention | Lightning Attention plus MLA |
| Layer mix | 10 MLA + 70 Lightning Attention |
| KV cache / token (bf16) | 11.2 KiB (Very low) |
| License | MIT License |

**Key detail:** Uses a 7:1 linear-attention/MLA ratio and a much larger 63B active path.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ling-2-5-1t.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A79-ant-groups-ling-25-1t-with-lightning-attention
- config.json: https://huggingface.co/inclusionAI/Ling-2.5-1T/blob/main/config.json
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ling-2-5-1t
- Web page: https://llm-atlas-7or.pages.dev/models/ling-2-5-1t/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ling-2-5-1t.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Tiny Aya (3.35B)

> Compact multilingual model from Cohere with a rare parallel transformer block.

| Property | Value |
| --- | --- |
| Lab | Cohere (Canada) |
| Released | 2026-02-13 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 3.35B parameters |
| Context window | 8,192 tokens |
| Attention | GQA with 3:1 sliding-window attention |
| Layer mix | 27 sliding-window + 9 global |
| KV cache / token (bf16) | 72 KiB (Low) |
| License | Creative Commons Attribution-NonCommercial 4.0 |

**Key detail:** Runs attention and the MLP in parallel while mixing RoPE with NoPE.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 4.8 (Scientific 2.6)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/tiny-aya-3-35b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A710-tiny-aya-a-335b-model-with-strong-multilingual-support
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/15_tiny-aya
- config.json: https://huggingface.co/CohereLabs/tiny-aya-base/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2603.11510
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-tiny-aya-3-35b
- Web page: https://llm-atlas-7or.pages.dev/models/tiny-aya-3-35b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/tiny-aya-3-35b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: MiniMax-M2.5 (230B)

> Popular 230B coder that opts for a classic architecture instead of the newer hybrid-attention ideas.

| Property | Value |
| --- | --- |
| Lab | MiniMax (China) |
| Released | 2026-02-12 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 230B total, 10B active (4.3% active) |
| Context window | 196,608 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 62 GQA |
| KV cache / token (bf16) | 248 KiB (High) |
| License | Modified MIT License |

**Key detail:** Deliberately avoids sliding-window or linear-attention hybrids while keeping a 10B active path and three MTP modules.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 22.8 (Scientific 10.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/minimax-m2-5-230b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A76-minimax-m25-a-strong-coder-with-only-230b-parameters
- config.json: https://huggingface.co/MiniMaxAI/MiniMax-M2.5/blob/main/config.json
- Tech report: https://arxiv.org/abs/2605.26494
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-minimax-m2-5-230b
- Web page: https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/minimax-m2-5-230b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-5 (744B)

> Huge GLM refresh that adopts both MLA and DeepSeek Sparse Attention for flagship-scale inference.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2026-02-11 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 744B total, 40B active (5.4% active) |
| Context window | 202,752 tokens |
| Attention | MLA with DeepSeek Sparse Attention |
| Layer mix | 78 MLA |
| KV cache / token (bf16) | 87.8 KiB (Moderate) |
| License | MIT License |

**Key detail:** Bigger than GLM-4.7, with more experts, fewer layers, and DeepSeek-style MTP.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 21.8 (Scientific 3.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-5-744b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A721-glm-5
- config.json: https://huggingface.co/zai-org/GLM-5/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2602.15763
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-5-744b
- Web page: https://llm-atlas-7or.pages.dev/models/glm-5-744b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-5-744b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nanbeige 4.1 (3B)

> Small on-device oriented model that stays close to Llama 3.2 while nudging the scaling choices.

| Property | Value |
| --- | --- |
| Lab | BOSS Zhipin (China) |
| Released | 2026-02-10 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 3B parameters |
| Context window | 262,144 tokens |
| Attention | GQA |
| Layer mix | 32 GQA |
| KV cache / token (bf16) | 64 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Llama-like stack without tying input embeddings to the output layer.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 8.4 (Scientific 5.4)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nanbeige-4-1-3b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A77-nanbeige-41-3b-a-strong-llama-3-successor
- config.json: https://huggingface.co/Nanbeige/Nanbeige4.1-3B/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2602.13367
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nanbeige-4-1-3b
- Web page: https://llm-atlas-7or.pages.dev/models/nanbeige-4-1-3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nanbeige-4-1-3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Step 3.5 Flash (196B)

> Throughput-oriented MoE model that stays competitive with much larger DeepSeek-style systems.

| Property | Value |
| --- | --- |
| Lab | StepFun (China) |
| Released | 2026-02-01 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 196B total, 11B active (5.6% active) |
| Context window | 262,144 tokens |
| Attention | GQA with 3:1 sliding-window attention |
| Layer mix | 36 sliding-window + 12 global |
| KV cache / token (bf16) | 192 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Uses MTP-3 during both training and inference for unusually high throughput.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 17 (Scientific 13.4)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/step-3-5-flash-196b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A73-stepfuns-step-35-flash-good-performance-at-great-tokens-sec-throughput
- config.json: https://huggingface.co/stepfun-ai/Step-3.5-Flash/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2602.10604
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-step-3-5-flash-196b
- Web page: https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/step-3-5-flash-196b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: LongCat-Flash-Lite (68.5B-A3B)

> Meituan's LongCat-Next text backbone is a lightweight MoE that shifts a large share of parameters into N-gram embeddings while keeping sparse inference around 3B active parameters.

| Property | Value |
| --- | --- |
| Lab | Meituan (China) |
| Released | 2026-01-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 68.5B total, ~3B active avg (4.4% active) |
| Context window | 256,000 tokens |
| Attention | MLA with RoPE + NoPE |
| Layer mix | 14 logical layers / 28 MLA sublayers |
| KV cache / token (bf16) | 31.5 KiB (Low) |
| License | MIT License |

**Key detail:** Uses N-gram embedding expansion, Shortcut MoE routing over 256 routed plus 128 zero/identity experts, and top-k 12 expert choices per token.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 11.5 (Scientific 2.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/longcat-flash-lite-68-5b-a3b.webp
- config.json: https://huggingface.co/meituan-longcat/LongCat-Flash-Lite/blob/main/config.json
- Tech report: https://arxiv.org/abs/2601.21204
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-longcat-flash-lite-68-5b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/longcat-flash-lite-68-5b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/longcat-flash-lite-68-5b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Arcee AI Trinity Large (400B)

> Arcee's flagship blends several efficiency tricks into a DeepSeek-like coarse MoE design; the AA score shown here refers to the Trinity Large Thinking checkpoint.

| Property | Value |
| --- | --- |
| Lab | Arcee AI (United States) |
| Released | 2026-01-27 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 400B total, 13B active (3.3% active) |
| Context window | 512,000 tokens |
| Attention | GQA with gated attention and 3:1 sliding-window/global attention |
| Layer mix | 45 sliding-window + 15 global |
| KV cache / token (bf16) | 240 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Combines QK-Norm, RoPE+NoPE, sandwich norm, and a coarse-grained MoE.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 10.8 (General 16.6, Scientific 8.4, Coding 20.6, Agents 0.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/arcee-ai-trinity-large-400b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A720-arcee-ai-trinity-large
- config.json: https://huggingface.co/arcee-ai/Trinity-Large-Base/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2602.17004
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-arcee-ai-trinity-large-400b
- Web page: https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/arcee-ai-trinity-large-400b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kimi K2.5 (1T)

> Native-multimodal Moonshot flagship that keeps the K2/DeepSeek-style MoE layout and pushes native context to 256k.

| Property | Value |
| --- | --- |
| Lab | Moonshot AI (China) |
| Released | 2026-01-27 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1T total, 32B active (3.2% active) |
| Context window | 256,000 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | Modified MIT License |

**Key detail:** Keeps the 384-expert K2 backbone, but adds multimodal capabilities (not shown) and doubles the native context length.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 19.4 (Scientific 6.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kimi-k2-5.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A72-moonshot-ais-kimi-k25-a-deepseek-like-model-at-a-1-trillion-parameter-scale
- config.json: https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/config.json
- License: https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2602.02276
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kimi-k2-5
- Web page: https://llm-atlas-7or.pages.dev/models/kimi-k2-5/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kimi-k2-5.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-4.7 (355B)

> Immediate GLM predecessor that stays closer to the older GLM-4.5 style before the MLA shift.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2025-12-22 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 355B total, 32B active (9% active) |
| Context window | 202,752 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 92 GQA |
| KV cache / token (bf16) | 368 KiB (Very high) |
| License | MIT License |

**Key detail:** Serves as the pre-MLA, pre-sparse-attention baseline and uses an MTP-3-style setup.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 17.4 (Scientific 3.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-4-7-355b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A721-glm-5
- config.json: https://huggingface.co/zai-org/GLM-4.7/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2508.06471
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-4-7-355b
- Web page: https://llm-atlas-7or.pages.dev/models/glm-4-7-355b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-4-7-355b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Xiaomi MiMo-V2-Flash (309B)

> Large MoE model that pushes sliding-window attention harder than most contemporaries.

| Property | Value |
| --- | --- |
| Lab | Xiaomi (China) |
| Released | 2025-12-16 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 309B total, 15B active (4.9% active) |
| Context window | 262,144 tokens |
| Attention | 5:1 sliding-window/global attention |
| Layer mix | 40 sliding-window + 8 global |
| KV cache / token (bf16) | 144 KiB (Moderate) |
| License | MIT License |

**Key detail:** Uses an unusually small 128-token local window plus multi-token prediction.

## Architecture concepts

- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 16 (Scientific 4.3)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/xiaomi-mimo-v2-flash-309b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A719-xiaomi-mimo-v2-flash
- config.json: https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2601.02780
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-xiaomi-mimo-v2-flash-309b
- Web page: https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-flash-309b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/xiaomi-mimo-v2-flash-309b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Nemotron 3 Nano (30B-A3B)

> NVIDIA's Nano model is the most extreme transformer-state-space hybrid in the gallery.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2025-12-04 |
| Decoder type | Hybrid MoE |
| Family | Hybrid MoE |
| Scale | 30B total, 3B active (10% active) |
| Context window | 1,000,000 tokens |
| Attention | Mostly Mamba-2 with a few GQA layers |
| Layer mix | 6 GQA + 23 Mamba-2 + 23 MoE |
| KV cache / token (bf16) | 6 KiB (Very low) |
| License | NVIDIA Nemotron Open Model License |

**Key detail:** Interleaves Mamba-2 and MoE blocks, using attention only sparingly.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6.8 (Scientific 2.3)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/nemotron-3-nano-30b-a3b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A7181-nemotron-3-nano
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16/blob/main/config.json
- License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/
- Tech report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Nano-Technical-Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-nemotron-3-nano-30b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-30b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/nemotron-3-nano-30b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Mistral Large 3 (673B)

> Mistral's new flagship effectively adopts the DeepSeek architecture and retunes the expert sizes.

| Property | Value |
| --- | --- |
| Lab | Mistral AI (France) |
| Released | 2025-12-02 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 673B total, 41B active (6.1% active) |
| Context window | 262,144 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Near-clone of DeepSeek V3 with larger experts, fewer routed experts, and multimodal support.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9.3 (General 17.1, Scientific 2.1, Coding 18.3, Agents 0.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mistral-3-large-673-billion.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A717-mistral-3-large
- params.json: https://huggingface.co/mistralai/Mistral-Large-3-675B-Instruct-2512/blob/main/params.json
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-mistral-large-3
- Web page: https://llm-atlas-7or.pages.dev/models/mistral-large-3/
- JSON: https://llm-atlas-7or.pages.dev/api/models/mistral-large-3.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: DeepSeek V3.2 (671B)

> DeepSeek's successor keeps the V3 template but adds sparse attention to cut long-context costs.

| Property | Value |
| --- | --- |
| Lab | DeepSeek (China) |
| Released | 2025-12-01 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 671B total, 37B active (5.5% active) |
| Context window | 128,000 tokens |
| Attention | MLA with DeepSeek Sparse Attention |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | MIT License |

**Key detail:** Adds sparse attention while retaining the DeepSeek V3-style MTP path.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 16 (Scientific 6)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/deepseek-v3-2-671b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A716-deepseek-v32
- config.json: https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/config.json
- License: https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2512.02556
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-deepseek-v3-2
- Web page: https://llm-atlas-7or.pages.dev/models/deepseek-v3-2/
- JSON: https://llm-atlas-7or.pages.dev/api/models/deepseek-v3-2.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: INTELLECT-3 (106B)

> Large-scale RL post-training of GLM-4.5-Air that keeps the compact 106B sparse MoE backbone intact.

| Property | Value |
| --- | --- |
| Lab | Prime Intellect (United States) |
| Released | 2025-11-26 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 106B total, 12B active (11.3% active) |
| Context window | 128,000 tokens |
| Attention | GQA |
| Layer mix | 46 GQA |
| KV cache / token (bf16) | 184 KiB (High) |
| License | MIT License |

**Key detail:** Keeps the GLM-4.5-Air architecture unchanged and shifts the capability profile through SFT plus large-scale RL.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 10.6 (Scientific 6.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/intellect-3.webp
- config.json: https://huggingface.co/PrimeIntellect/INTELLECT-3/blob/main/config.json
- Tech report: https://storage.googleapis.com/intellect-3-paper/INTELLECT_3_Technical_Report.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-intellect-3
- Web page: https://llm-atlas-7or.pages.dev/models/intellect-3/
- JSON: https://llm-atlas-7or.pages.dev/api/models/intellect-3.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: OLMo 3 (32B)

> Scaled-up OLMo 3 keeps the same block design but moves to grouped-query attention.

| Property | Value |
| --- | --- |
| Lab | Ai2 (United States) |
| Released | 2025-11-20 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 32B parameters |
| Context window | 65,536 tokens |
| Attention | GQA with QK-Norm and 3:1 sliding-window/global attention |
| Layer mix | 48 sliding-window + 16 global |
| KV cache / token (bf16) | 256 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Keeps post-norm while scaling width and applying YaRN only on global layers.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6.5 (Scientific 3.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/olmo-3-32b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A715-olmo-3-thinking
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/13_olmo3
- config.json: https://huggingface.co/allenai/Olmo-3-32B-Think/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2512.13961
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-olmo-3-32b
- Web page: https://llm-atlas-7or.pages.dev/models/olmo-3-32b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/olmo-3-32b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: OLMo 3 (7B)

> New transparent Allen AI model that keeps OLMo's post-norm flavor while modernizing context handling.

| Property | Value |
| --- | --- |
| Lab | Ai2 (United States) |
| Released | 2025-11-20 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 7B parameters |
| Context window | 65,536 tokens |
| Attention | MHA with QK-Norm and 3:1 sliding-window/global attention |
| Layer mix | 24 sliding-window + 8 global |
| KV cache / token (bf16) | 512 KiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Retains post-norm, keeps MHA, and applies YaRN only on global layers.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.2 (Scientific 2.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/olmo-3-7b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A715-olmo-3-thinking
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/13_olmo3
- config.json: https://huggingface.co/allenai/Olmo-3-1025-7B/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2512.13961
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-olmo-3-7b
- Web page: https://llm-atlas-7or.pages.dev/models/olmo-3-7b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/olmo-3-7b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kimi Linear (48B-A3B)

> Linear-attention hybrid that keeps a transformer backbone but replaces most full-attention layers.

| Property | Value |
| --- | --- |
| Lab | Moonshot AI (China) |
| Released | 2025-10-30 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 48B total, 3B active (6.3% active) |
| Context window | 1,000,000 tokens |
| Attention | 3:1 Kimi Delta Attention and MLA |
| Layer mix | 7 MLA + 20 Kimi Delta Attention |
| KV cache / token (bf16) | 7.9 KiB (Very low) |
| License | MIT License |

**Key detail:** Uses NoPE in MLA layers, channel-wise KDA gating, and kernel-4 ShortConv preprocessing for Q, K, and V in KDA layers.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Short Convolution (ShortConv)](https://llm-atlas-7or.pages.dev/concepts/shortconv.md): A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.
- [Attention Residuals (AttnRes)](https://llm-atlas-7or.pages.dev/concepts/attention-residuals.md): Learned, attention-style weights over earlier layer outputs replace the plain residual sum.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 7.3

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kimi-linear-48b-a3b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A7144-kimi-linear-vs-qwen3-next
- config.json: https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Base/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2510.26692
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kimi-linear-48b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kimi-linear-48b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Ouro-Thinking (2.6B)

> ByteDance's dense LoopLM reuses one 48-layer MHA stack across four recurrent passes and trains an exit gate over the intermediate representations.

| Property | Value |
| --- | --- |
| Lab | ByteDance (China) |
| Released | 2025-10-29 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 2.6B parameters |
| Context window | 65,536 tokens |
| Attention | MHA with RoPE |
| Layer mix | 48 shared MHA layers × 4 passes |
| KV cache / token (bf16) | 1.5 MiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Reuses the same 48-layer MHA stack across four passes, yielding 192 block applications, with sandwich RMSNorm and a learned exit gate.

## Architecture concepts

- [Looped Transformer (Looped depth)](https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing.md): The same blocks are applied several times, reusing weights across depth.
- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/ouro-2-6b.webp
- config.json: https://huggingface.co/ByteDance/Ouro-2.6B-Thinking/blob/main/config.json
- License: https://huggingface.co/ByteDance/Ouro-2.6B-Thinking/blob/main/README.md#license
- Tech report: https://arxiv.org/abs/2510.25741
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-ouro-thinking-2-6b
- Web page: https://llm-atlas-7or.pages.dev/models/ouro-thinking-2-6b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/ouro-thinking-2-6b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: MiniMax M2 (230B)

> MiniMax's flagship returns to full attention and looks like a leaner, sparser cousin of Qwen3.

| Property | Value |
| --- | --- |
| Lab | MiniMax (China) |
| Released | 2025-10-23 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 230B total, 10B active (4.3% active) |
| Context window | 196,608 tokens |
| Attention | GQA with QK-Norm and partial RoPE |
| Layer mix | 62 GQA |
| KV cache / token (bf16) | 248 KiB (High) |
| License | Modified MIT License |

**Key detail:** Uses per-layer QK-Norm, much sparser MoE routing than Qwen3, and MTP during training.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 18.6 (Scientific 7.3)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/minimax-m2-230b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A7131-per-layer-qk-norm
- config.json: https://huggingface.co/MiniMaxAI/MiniMax-M2/blob/main/config.json
- Tech report: https://arxiv.org/abs/2605.26494
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-minimax-m2-230b
- Web page: https://llm-atlas-7or.pages.dev/models/minimax-m2-230b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/minimax-m2-230b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 Next (80B-A3B)

> Efficiency-focused Qwen refresh that swaps standard attention for a DeltaNet-attention hybrid.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-09-09 |
| Decoder type | Sparse hybrid |
| Family | Hybrid MoE |
| Scale | 80B total, 3B active (3.8% active) |
| Context window | 262,144 tokens |
| Attention | 3:1 Gated DeltaNet and Gated Attention |
| Layer mix | 12 gated attention + 36 DeltaNet |
| KV cache / token (bf16) | 24 KiB (Very low) |
| License | Apache License 2.0 |

**Key detail:** Adds many more experts, a shared expert, native 262k context, and MTP for speculative decoding.

## Architecture concepts

- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Hybrid Attention (Hybrid attention)](https://llm-atlas-7or.pages.dev/concepts/hybrid-attention.md): Mixes cheap linear-attention or state-space layers with a few full-attention layers.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9.6 (Scientific 3.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-next-80b-a3b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A7121-expert-size-and-number
- config.json: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-next-80b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-next-80b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-next-80b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Grok 2.5 (270B)

> Rare production-model release that shows an older MoE style with fewer, larger experts.

| Property | Value |
| --- | --- |
| Lab | xAI (United States) |
| Released | 2025-08-22 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 270B parameters |
| Context window | 131,072 tokens |
| Attention | GQA |
| Layer mix | 64 GQA |
| KV cache / token (bf16) | 256 KiB (High) |
| License | Grok 2 Community License Agreement |

**Key detail:** Adds an always-on SwiGLU path that effectively behaves like a shared expert.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/grok-2-5-270b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A710-grok-25
- config.json: https://huggingface.co/xai-org/grok-2/blob/main/config.json
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-grok-2-5-270b
- Web page: https://llm-atlas-7or.pages.dev/models/grok-2-5-270b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/grok-2-5-270b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 3 (270M)

> Tiny Gemma 3 variant that preserves the family's local-global attention recipe at a toy scale.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2025-08-14 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 270M parameters |
| Context window | 128,000 tokens |
| Attention | Multi-query attention with QK-Norm and 5:1 sliding-window/global attention |
| Layer mix | 15 sliding-window + 3 global |
| KV cache / token (bf16) | 18 KiB (Very low) |
| License | Gemma Terms of Use + Gemma Prohibited Use Policy |

**Key detail:** Keeps the Gemma 3 stack shape while shrinking down to 4 attention heads, a single KV head, and the same 262k vocabulary.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.1 (Scientific 1.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-3-270m.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A74-mistral-small-31
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/12_gemma3
- config.json: https://huggingface.co/google/gemma-3-270m/blob/main/config.json
- License: https://ai.google.dev/gemma/prohibited_use_policy
- Tech report: https://arxiv.org/pdf/2503.19786
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-3-270m
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-3-270m/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-3-270m.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GPT-OSS (120B)

> Larger gpt-oss variant keeps the same alternating-attention recipe as the 20B model.

| Property | Value |
| --- | --- |
| Lab | OpenAI (United States) |
| Released | 2025-08-04 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 117B total, 5.1B active (4.4% active) |
| Context window | 128,000 tokens |
| Attention | GQA with alternating sliding-window and global layers |
| Layer mix | 18 sliding-window + 18 global |
| KV cache / token (bf16) | 72 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Shared architectural template scaled up for OpenAI's flagship open-weight release.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 11.6 (General 18.8, Scientific 10.4, Coding 17, Agents 1.8)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gpt-oss-120b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A79-gpt-oss
- config.json: https://huggingface.co/openai/gpt-oss-120b/blob/main/config.json
- License: https://huggingface.co/openai/gpt-oss-120b/blob/main/LICENSE
- Tech report: https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gpt-oss-120b
- Web page: https://llm-atlas-7or.pages.dev/models/gpt-oss-120b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gpt-oss-120b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GPT-OSS (20B)

> OpenAI's smaller open-weight MoE model favors width and alternating local/global attention.

| Property | Value |
| --- | --- |
| Lab | OpenAI (United States) |
| Released | 2025-08-04 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 21B total, 3.6B active (17.1% active) |
| Context window | 128,000 tokens |
| Attention | GQA with alternating sliding-window and global layers |
| Layer mix | 12 sliding-window + 12 global |
| KV cache / token (bf16) | 48 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Wider and shallower than Qwen3, with attention bias and sink mechanisms.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9 (General 12.8, Scientific 6.2, Coding 19.4, Agents 0)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gpt-oss-20b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A79-gpt-oss
- config.json: https://huggingface.co/openai/gpt-oss-20b/blob/main/config.json
- License: https://huggingface.co/openai/gpt-oss-20b/blob/main/LICENSE
- Tech report: https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gpt-oss-20b
- Web page: https://llm-atlas-7or.pages.dev/models/gpt-oss-20b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gpt-oss-20b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 Coder Flash (30B-A3B)

> Coding-tuned Qwen model that keeps a straightforward grouped-query MoE stack instead of the newer hybrid-attention variants.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-07-31 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 30B total, 3.3B active (11% active) |
| Context window | 256,000 tokens |
| Attention | GQA |
| Layer mix | 48 GQA |
| KV cache / token (bf16) | 96 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Uses 128 experts with 8 active per token and a native 256k context window for coding workloads.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9.6 (Scientific 1.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-coder-flash-30b-a3b-mixture-of-experts.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A74-qwen3-coder-next-an-attention-hybrid-for-coding
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11_qwen3
- config.json: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-coder-flash-30b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-coder-flash-30b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-coder-flash-30b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-4.5 (355B)

> Agent-oriented instruction/reasoning hybrid that borrows DeepSeek's dense-prefix MoE layout.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2025-07-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 355B total, 32B active (9% active) |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 92 GQA |
| KV cache / token (bf16) | 368 KiB (Very high) |
| License | MIT License |

**Key detail:** Starts with three dense layers before MoE routing, keeps a shared expert, and uses MTP during training.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 12.8 (Scientific 6.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-4-5-355b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A711-glm-45
- config.json: https://huggingface.co/zai-org/GLM-4.5/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2508.06471
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-4-5-355b
- Web page: https://llm-atlas-7or.pages.dev/models/glm-4-5-355b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-4-5-355b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GLM-4.5-Air (106B)

> Compact GLM-4.5 companion that keeps the same agent-oriented sparse MoE recipe at a smaller serving footprint.

| Property | Value |
| --- | --- |
| Lab | Z.ai (China) |
| Released | 2025-07-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 106B total, 12B active (11.3% active) |
| Context window | 128,000 tokens |
| Attention | GQA |
| Layer mix | 46 GQA |
| KV cache / token (bf16) | 184 KiB (High) |
| License | MIT License |

**Key detail:** Shrinks the GLM-4.5 layout to 46 layers, keeps the dense warmup, and inherits the MTP training recipe.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 11.1 (Scientific 3.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/glm-4-5-air.webp
- config.json: https://huggingface.co/zai-org/GLM-4.5-Air/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2508.06471
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-glm-4-5-air
- Web page: https://llm-atlas-7or.pages.dev/models/glm-4-5-air/
- JSON: https://llm-atlas-7or.pages.dev/api/models/glm-4-5-air.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Kimi K2 (1T)

> Trillion-parameter Moonshot model that essentially scales the DeepSeek V3 recipe upward.

| Property | Value |
| --- | --- |
| Lab | Moonshot AI (China) |
| Released | 2025-07-10 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 1T total, 32B active (3.2% active) |
| Context window | 128,000 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | Modified MIT License |

**Key detail:** More experts and fewer MLA heads than DeepSeek V3.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 12.7 (Scientific 3.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/kimi-k2-1-trillion.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A78-kimi-k2-and-kimi-k2-thinking
- config.json: https://huggingface.co/moonshotai/Kimi-K2-Base/blob/main/config.json
- License: https://huggingface.co/moonshotai/Kimi-K2-Base/blame/main/LICENSE
- Tech report: https://arxiv.org/pdf/2507.20534
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-kimi-k2
- Web page: https://llm-atlas-7or.pages.dev/models/kimi-k2/
- JSON: https://llm-atlas-7or.pages.dev/api/models/kimi-k2.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: SmolLM3 (3B)

> Compact dense model that experiments with leaving out positional encodings in selected layers.

| Property | Value |
| --- | --- |
| Lab | Hugging Face (United States) |
| Released | 2025-06-19 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 3B parameters |
| Context window | 131,072 tokens |
| Attention | GQA with periodic NoPE layers |
| Layer mix | 36 GQA |
| KV cache / token (bf16) | 72 KiB (Low) |
| License | Apache License 2.0 |

**Key detail:** Every fourth layer omits RoPE to test a NoPE-style cadence.

## Architecture concepts

- [No Positional Embeddings (NoPE)](https://llm-atlas-7or.pages.dev/concepts/nope.md): Some or all layers drop explicit positional encodings and rely on the causal mask.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/smollm3-3b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A77-smollm3
- config.json: https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base/blob/main/config.json
- Tech report: https://huggingface.co/blog/smollm3
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-smollm3-3b
- Web page: https://llm-atlas-7or.pages.dev/models/smollm3-3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/smollm3-3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 (0.6B)

> Tiny current-generation Qwen model that trades width for more depth and a low memory footprint.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-04-28 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 0.6B parameters |
| Context window | 32,768 tokens |
| Attention | GQA |
| Layer mix | 28 GQA |
| KV cache / token (bf16) | 112 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Deeper than Llama 3.2 1B and unusually practical for local experiments and teaching.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 4.8 (Scientific 2.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-0-6b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A761-qwen3-dense
- config.json: https://huggingface.co/Qwen/Qwen3-0.6B-Base/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-0.6B-Base/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-0-6b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-0-6b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-0-6b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 (235B-A22B)

> Large sparse Qwen variant that stays very close to DeepSeek V3 while removing the shared expert.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-04-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 235B total, 22B active (9.4% active) |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 94 GQA |
| KV cache / token (bf16) | 188 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** High-capacity MoE design optimized for serving efficiency without a shared expert.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 8.3 (Scientific 2.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-235b-a22b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A762-qwen3-moe
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11_qwen3
- config.json: https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2505.09388
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-235b-a22b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-235b-a22b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-235b-a22b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 (30B-A3B)

> Small sparse Qwen3 model that sits close to GPT-OSS in active size but uses a deeper stack.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-04-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 30B total, 3B active (10% active) |
| Context window | 128,000 tokens |
| Attention | GQA |
| Layer mix | 48 GQA |
| KV cache / token (bf16) | 96 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Deeper, narrower MoE alternative to GPT-OSS without a shared expert.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6.6 (Scientific 2.3)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-30b-a3b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A79-gpt-oss
- config.json: https://huggingface.co/Qwen/Qwen3-30B-A3B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-30B-A3B/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-30b-a3b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-30b-a3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-30b-a3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 (32B)

> Large dense Qwen3 model that serves as the clearest like-for-like comparison for OLMo 3 32B.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-04-28 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 32B parameters |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 64 GQA |
| KV cache / token (bf16) | 256 KiB (High) |
| License | Apache License 2.0 |

**Key detail:** Reference dense Qwen stack with QK-Norm and 8 KV heads.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 7.3

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-32b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A715-olmo-3-thinking
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11_qwen3
- config.json: https://huggingface.co/Qwen/Qwen3-32B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-32B/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2505.09388
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-32b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-32b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-32b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 (4B)

> Mid-size dense Qwen3 model used here as a clean baseline against SmolLM3 and Tiny Aya.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-04-28 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 4B parameters |
| Context window | 32,768 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 36 GQA |
| KV cache / token (bf16) | 144 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Compact Qwen3 dense stack with QK-Norm and a 151k vocabulary.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6.6

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-4b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A77-smollm3
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11_qwen3
- config.json: https://huggingface.co/Qwen/Qwen3-4B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-4B/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2505.09388
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-4b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-4b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-4b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Qwen3 (8B)

> Dense Qwen3 baseline used here to show how little OLMo 3 changed the overall decoder recipe.

| Property | Value |
| --- | --- |
| Lab | Alibaba Cloud (China) |
| Released | 2025-04-28 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 8B parameters |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm |
| Layer mix | 36 GQA |
| KV cache / token (bf16) | 144 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Reference Qwen3 dense stack with QK-Norm and 8 KV heads.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 6 (Scientific 1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/qwen3-8b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A715-olmo-3-thinking
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11_qwen3
- config.json: https://huggingface.co/Qwen/Qwen3-8B/blob/main/config.json
- License: https://huggingface.co/Qwen/Qwen3-8B/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2505.09388
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-qwen3-8b
- Web page: https://llm-atlas-7or.pages.dev/models/qwen3-8b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/qwen3-8b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Llama 4 Maverick (400B)

> Meta's large MoE follows the DeepSeek V3 playbook but with a more conventional attention stack.

| Property | Value |
| --- | --- |
| Lab | Meta (United States) |
| Released | 2025-04-05 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 400B total, 17B active (4.3% active) |
| Context window | 1,000,000 tokens |
| Attention | GQA |
| Layer mix | 36 chunked + 12 full GQA |
| KV cache / token (bf16) | 192 KiB (High) |
| License | Llama 4 Community License Agreement |

**Key detail:** Alternates dense and MoE blocks and uses fewer, larger experts than DeepSeek V3.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 9.3 (General 18.8, Scientific 2.5, Coding 15.9, Agents 0)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/llama-4-maverick-400b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A75-llama-4
- config.json: https://huggingface.co/meta-llama/Llama-4-Maverick-17B-128E-Instruct/blob/main/config.json
- Tech report: https://ai.meta.com/blog/llama-4-multimodal-intelligence/
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-llama-4-maverick
- Web page: https://llm-atlas-7or.pages.dev/models/llama-4-maverick/
- JSON: https://llm-atlas-7or.pages.dev/api/models/llama-4-maverick.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Mistral Small 3.1 (24B)

> Fast dense 24B model that drops the sliding-window setup used in older Mistral releases.

| Property | Value |
| --- | --- |
| Lab | Mistral AI (France) |
| Released | 2025-03-18 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 24B parameters |
| Context window | 128,000 tokens |
| Attention | Standard GQA |
| Layer mix | 40 GQA |
| KV cache / token (bf16) | 160 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Latency-focused design with a smaller KV cache and fewer layers than Gemma 3 27B.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 7.1 (General 12.8, Scientific 2.2, Coding 13.9, Agents 0.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/mistral-3-1-small-24b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A74-mistral-small-31
- config.json: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503/blob/main/config.json
- Tech report: https://mistral.ai/news/mistral-small-3-1
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-mistral-small-3-1-24b
- Web page: https://llm-atlas-7or.pages.dev/models/mistral-small-3-1-24b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/mistral-small-3-1-24b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: xLSTM (7B)

> Recurrent 7B language model that replaces self-attention with xLSTM blocks built around matrix memory.

| Property | Value |
| --- | --- |
| Lab | NXAI (Austria) |
| Released | 2025-03-17 |
| Decoder type | Recurrent |
| Family | Recurrent |
| Scale | 7B parameters |
| Context window | No explicit limit tokens |
| Attention | No self-attention; mLSTM recurrent layers with matrix memory |
| Layer mix | 32 mLSTM |
| KV cache / token (bf16) | 0 B (No cache) |
| License | NXAI Community License Agreement |

**Key detail:** Stateful recurrent architecture aimed at fast long-context inference without an explicit context window.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/xlstm-7b.webp
- config.json: https://huggingface.co/NX-AI/xLSTM-7b/blob/main/config.json
- License: https://huggingface.co/NX-AI/xLSTM-7b/blob/main/LICENSE
- Tech report: https://arxiv.org/abs/2503.13427
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-xlstm-7b
- Web page: https://llm-atlas-7or.pages.dev/models/xlstm-7b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/xlstm-7b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Gemma 3 (27B)

> Gemma's flagship text stack leans on local attention more aggressively than Gemma 2.

| Property | Value |
| --- | --- |
| Lab | Google (United States) |
| Released | 2025-03-11 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 27B parameters |
| Context window | 128,000 tokens |
| Attention | GQA with QK-Norm and 5:1 sliding-window/global attention |
| Layer mix | 52 sliding-window + 10 global |
| KV cache / token (bf16) | 496 KiB (Very high) |
| License | Gemma Terms of Use + Gemma Prohibited Use Policy |

**Key detail:** Built around a 27B sweet spot with heavier local attention and a large 262k multilingual vocabulary.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 4.9 (General 6.9, Scientific 2.2, Coding 11.6, Agents 0)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gemma-3-27b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A74-mistral-small-31
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/12_gemma3
- config.json: https://huggingface.co/google/gemma-3-27b-it/blob/main/config.json
- License: https://ai.google.dev/gemma/prohibited_use_policy
- Tech report: https://arxiv.org/pdf/2503.19786
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gemma-3-27b
- Web page: https://llm-atlas-7or.pages.dev/models/gemma-3-27b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gemma-3-27b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: DeepSeek R1 (671B)

> Reasoning-tuned DeepSeek model built on the V3 architecture rather than a new base design.

| Property | Value |
| --- | --- |
| Lab | DeepSeek (China) |
| Released | 2025-01-20 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 671B total, 37B active (5.5% active) |
| Context window | 128,000 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | MIT License |

**Key detail:** Architecture matches DeepSeek V3, including the MTP-1 path; the main change is the reasoning-oriented training recipe.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 11.4 (General 22.2, Scientific 4.5, Coding 19.2, Agents 0)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/deepseek-v3-r1-671-billion.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A78-kimi-k2-and-kimi-k2-thinking
- config.json: https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/config.json
- License: https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2501.12948
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-deepseek-r1
- Web page: https://llm-atlas-7or.pages.dev/models/deepseek-r1/
- JSON: https://llm-atlas-7or.pages.dev/api/models/deepseek-r1.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: DeepSeek V3 (671B)

> DeepSeek's flagship template kicked off the recent wave of large open MoE models.

| Property | Value |
| --- | --- |
| Lab | DeepSeek (China) |
| Released | 2024-12-26 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 671B total, 37B active (5.5% active) |
| Context window | 128,000 tokens |
| Attention | MLA |
| Layer mix | 61 MLA |
| KV cache / token (bf16) | 68.6 KiB (Low) |
| License | DeepSeek License Agreement v1.0 |

**Key detail:** Uses a dense prefix, shared expert, and MTP-1 objective to keep a very large model practical.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 8.5 (General 15.3, Scientific 1.5, Coding 17.9, Agents 0.1)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/deepseek-v3-r1-671-billion.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A75-llama-4
- config.json: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json
- License: https://huggingface.co/deepseek-ai/DeepSeek-V3/blame/main/LICENSE-MODEL
- Tech report: https://arxiv.org/pdf/2412.19437
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-deepseek-v3
- Web page: https://llm-atlas-7or.pages.dev/models/deepseek-v3/
- JSON: https://llm-atlas-7or.pages.dev/api/models/deepseek-v3.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Phi-4 (14B)

> Microsoft's 14B dense Phi refresh stays close to Phi-3-medium but swaps its sliding-window attention for full-context GQA and a larger tokenizer.

| Property | Value |
| --- | --- |
| Lab | Microsoft (United States) |
| Released | 2024-12-12 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 14B parameters |
| Context window | 16,384 tokens |
| Attention | GQA with RoPE |
| Layer mix | 40 GQA |
| KV cache / token (bf16) | 200 KiB (High) |
| License | MIT License |

**Key detail:** Classic pre-norm RMSNorm stack with GQA, 40 heads, 10 KV heads, and a 100,352-token vocabulary.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.9 (Scientific 1.9)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/phi-4.webp
- config.json: https://huggingface.co/microsoft/phi-4/blob/main/config.json
- License: https://huggingface.co/microsoft/phi-4/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2412.08905
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-phi-4
- Web page: https://llm-atlas-7or.pages.dev/models/phi-4/
- JSON: https://llm-atlas-7or.pages.dev/api/models/phi-4.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: OLMo 2 (7B)

> Transparent dense model that keeps classic MHA and pushes normalization changes for training stability.

| Property | Value |
| --- | --- |
| Lab | Ai2 (United States) |
| Released | 2024-11-25 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 7B parameters |
| Context window | 4,096 tokens |
| Attention | MHA with QK-Norm |
| Layer mix | 32 MHA |
| KV cache / token (bf16) | 512 KiB (Very high) |
| License | Apache License 2.0 |

**Key detail:** Uses inside-residual post-norm instead of the usual pre-norm layout.

## Architecture concepts

- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.
- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.6

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/olmo-2-7b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A723-olmo-2-summary
- config.json: https://huggingface.co/allenai/OLMo-2-1124-7B-Instruct/blob/main/config.json
- Tech report: https://arxiv.org/pdf/2501.00656
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-olmo-2-7b
- Web page: https://llm-atlas-7or.pages.dev/models/olmo-2-7b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/olmo-2-7b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Llama 3.2 (1B)

> Small dense Llama baseline in the Qwen comparison, with fewer layers but more width.

| Property | Value |
| --- | --- |
| Lab | Meta (United States) |
| Released | 2024-09-25 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 1B parameters |
| Context window | 128,000 tokens |
| Attention | GQA |
| Layer mix | 16 GQA |
| KV cache / token (bf16) | 32 KiB (Low) |
| License | Llama Community License Agreement (variant-specific) |

**Key detail:** Wider architecture with more heads than Qwen3 0.6B.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 4.8 (Scientific 2.7)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/llama-3-2-1b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A761-qwen3-dense
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/07_gpt_to_llama
- License: https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/LICENSE.txt
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-llama-3-2-1b
- Web page: https://llm-atlas-7or.pages.dev/models/llama-3-2-1b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/llama-3-2-1b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Llama 3.2 (3B)

> Small Llama baseline used to show how closely Nanbeige follows the mainstream compact decoder recipe.

| Property | Value |
| --- | --- |
| Lab | Meta (United States) |
| Released | 2024-09-25 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 3B parameters |
| Context window | 128,000 tokens |
| Attention | GQA |
| Layer mix | 28 GQA |
| KV cache / token (bf16) | 112 KiB (Moderate) |
| License | Llama 3.2 Community License Agreement |

**Key detail:** Reference small-model Llama architecture with tied embeddings.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 5.7

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/llama-3-2-3b.webp
- View in article: https://magazine.sebastianraschka.com/p/a-dream-of-spring-for-open-weight#%C2%A77-nanbeige-41-3b-a-strong-llama-3-successor
- License: https://huggingface.co/meta-llama/Llama-3.2-3B/blob/main/LICENSE.txt
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-llama-3-2-3b
- Web page: https://llm-atlas-7or.pages.dev/models/llama-3-2-3b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/llama-3-2-3b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: Llama 3 (8B)

> Reference dense Llama stack used to contrast OLMo 2's normalization and attention choices.

| Property | Value |
| --- | --- |
| Lab | Meta (United States) |
| Released | 2024-04-18 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 8B parameters |
| Context window | 8,192 tokens |
| Attention | GQA with RoPE |
| Layer mix | 32 GQA |
| KV cache / token (bf16) | 128 KiB (Moderate) |
| License | Llama 3 Community License Agreement |

**Key detail:** Pre-norm baseline; wider than OLMo 2 at a similar scale.

## Architecture concepts

- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 4.8 (Scientific 2.5)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/llama-3-8b.webp
- View in article: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison#%C2%A723-olmo-2-summary
- From scratch: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/07_gpt_to_llama
- config.json: https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/config.json
- License: https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE
- Tech report: https://arxiv.org/pdf/2407.21783
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-llama-3-8b
- Web page: https://llm-atlas-7or.pages.dev/models/llama-3-8b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/llama-3-8b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

## Model: GPT-2 XL (1.5B)

> Late-2019 dense baseline included here as a reference point for how much decoder stacks have changed since GPT-2.

| Property | Value |
| --- | --- |
| Lab | OpenAI (United States) |
| Released | 2019-11-05 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 1.5B parameters |
| Context window | 1,024 tokens |
| Attention | MHA with learned absolute positional embeddings |
| Layer mix | 48 MHA |
| KV cache / token (bf16) | 300 KiB (High) |
| License | OpenAI "Modified MIT" license |

**Key detail:** Classic GPT-2 recipe with dropout, GELU, LayerNorm, and full multi-head attention.

## Architecture concepts

- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gpt-2-xl.webp
- View in article: https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzing-the#%C2%A72-coming-from-gpt-2
- config.json: https://huggingface.co/openai-community/gpt2-xl/blob/main/config.json
- Tech report: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gpt-2-xl-1-5b
- Web page: https://llm-atlas-7or.pages.dev/models/gpt-2-xl-1-5b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gpt-2-xl-1-5b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._

---

## Concept: Multi-Head Attention (MHA)

> The original transformer attention: every head has its own queries, keys and values.

Category: Attention

Self-attention lets every token build its representation from a weighted mix of all earlier tokens. The weights come from comparing a token's query with every key; the values are what gets mixed.

Multi-head attention runs several of these attention operations side by side, each in a smaller subspace, so different heads can track different relationships (syntax, coreference, position) at once. Their outputs are concatenated and projected back.

In a decoder, a causal mask hides future tokens. Because every head stores its own keys and values, MHA has the largest KV cache of the common variants.

**Trade-off:** Maximum expressiveness per layer, but the KV cache grows with heads × layers × tokens.

## Models using MHA (4)

- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md) — Ai2, 2025-11-20
- [Ouro-Thinking (2.6B)](https://llm-atlas-7or.pages.dev/models/ouro-thinking-2-6b.md) — ByteDance, 2025-10-29
- [OLMo 2 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-2-7b.md) — Ai2, 2024-11-25
- [GPT-2 XL (1.5B)](https://llm-atlas-7or.pages.dev/models/gpt-2-xl-1-5b.md) — OpenAI, 2019-11-05

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/mha/
Web page: https://llm-atlas-7or.pages.dev/concepts/mha/

## Concept: Grouped-Query Attention (GQA)

> Several query heads share one key/value head, shrinking the KV cache with little quality loss.

Category: Attention

GQA keeps many query heads but groups them so each group reads from a single shared key/value head. With 32 query heads and 8 KV heads, the cache is four times smaller than MHA.

Multi-query attention (MQA) is the extreme case with one KV head for everyone; GQA sits between MQA and MHA and became the default for dense open models from Llama 2 onwards.

It is simple to implement, plays well with existing kernels, and is often combined with sliding-window layers for further savings.

**Trade-off:** Large memory and bandwidth savings at inference for a small modelling cost.

## Models using GQA (78)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [MiniMax M3.1-Flash (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-1-flash-428b.md) — MiniMax, 2026-10-19
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md) — NVIDIA, 2026-10-15
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Naive-N0.5-Flash (309B)](https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b.md) — NaiveAI, 2026-09-27
- [Xiaomi MiMo-V2.6-Flash-RL (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b.md) — Xiaomi, 2026-09-21
- [Xiaomi MiMo-V2.6-Pro-RL (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t.md) — Xiaomi, 2026-09-21
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md) — Alibaba Cloud, 2026-08-14
- [Nemotron 3.5 Lightning (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-lightning-30b-a3b.md) — NVIDIA, 2026-08-11
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md) — Upstage, 2026-07-22
- [Antares (1B)](https://llm-atlas-7or.pages.dev/models/antares-1b.md) — Cisco, 2026-07-21
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [Nanbeige 4.2 (3B)](https://llm-atlas-7or.pages.dev/models/nanbeige-4-2-3b.md) — BOSS Zhipin, 2026-07-21
- [BTL-3 (27B)](https://llm-atlas-7or.pages.dev/models/btl-3-27b.md) — Bad Theory Labs, 2026-07-20
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Soofi S (30B-A3B)](https://llm-atlas-7or.pages.dev/models/soofi-s-30b-a3b.md) — SOOFI Consortium, 2026-07-10
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [VibeThinker-3B (3B)](https://llm-atlas-7or.pages.dev/models/vibethinker-3b.md) — Weibo, 2026-06-15
- [MiniMax M3 (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-428b.md) — MiniMax, 2026-06-13
- [North Mini Code (30B-A3B)](https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b.md) — Cohere, 2026-06-05
- [Nemotron 3 Ultra (550B-A55B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-ultra-550b-a55b.md) — NVIDIA, 2026-06-04
- [Gemma 4 (12B)](https://llm-atlas-7or.pages.dev/models/gemma-4-12b.md) — Google, 2026-06-03
- [JetBrains Mellum2 Thinking (12B-A2.5B)](https://llm-atlas-7or.pages.dev/models/jetbrains-mellum2-thinking-12b-a2-5b.md) — JetBrains, 2026-06-01
- [LFM2.5 (1.2B)](https://llm-atlas-7or.pages.dev/models/lfm2-5-1-2b.md) — Liquid AI, 2026-05-28
- [LFM2.5 (350M)](https://llm-atlas-7or.pages.dev/models/lfm2-5-350m.md) — Liquid AI, 2026-05-28
- [LFM2.5 (8B-A1B)](https://llm-atlas-7or.pages.dev/models/lfm2-5-8b-a1b.md) — Liquid AI, 2026-05-28
- [Command A+ (218B-A25B)](https://llm-atlas-7or.pages.dev/models/command-a-218b-a25b.md) — Cohere, 2026-05-20
- [ZAYA1-8B (8.4B)](https://llm-atlas-7or.pages.dev/models/zaya1-8b.md) — Zyphra, 2026-05-06
- [Granite 4.1 (30B)](https://llm-atlas-7or.pages.dev/models/granite-4-1-30b.md) — IBM, 2026-04-29
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28
- [Tencent Hy3-preview (295B-A21B)](https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b.md) — Tencent, 2026-04-23
- [Xiaomi MiMo-V2.5-Pro (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-pro-1-02t.md) — Xiaomi, 2026-04-22
- [Gemma 4 (31B)](https://llm-atlas-7or.pages.dev/models/gemma-4-31b.md) — Google, 2026-04-02
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02
- [MiniMax M2.7 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b.md) — MiniMax, 2026-03-18
- [Nemotron 3 Nano (4B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-4b.md) — NVIDIA, 2026-03-16
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md) — NVIDIA, 2026-03-11
- [Sarvam (30B)](https://llm-atlas-7or.pages.dev/models/sarvam-30b.md) — Sarvam AI, 2026-03-03
- [Sarvam (105B)](https://llm-atlas-7or.pages.dev/models/sarvam-105b.md) — Sarvam AI, 2026-03-03
- [Tiny Aya (3.35B)](https://llm-atlas-7or.pages.dev/models/tiny-aya-3-35b.md) — Cohere, 2026-02-13
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [Nanbeige 4.1 (3B)](https://llm-atlas-7or.pages.dev/models/nanbeige-4-1-3b.md) — BOSS Zhipin, 2026-02-10
- [Step 3.5 Flash (196B)](https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b.md) — StepFun, 2026-02-01
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [Nemotron 3 Nano (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-30b-a3b.md) — NVIDIA, 2025-12-04
- [INTELLECT-3 (106B)](https://llm-atlas-7or.pages.dev/models/intellect-3.md) — Prime Intellect, 2025-11-26
- [OLMo 3 (32B)](https://llm-atlas-7or.pages.dev/models/olmo-3-32b.md) — Ai2, 2025-11-20
- [MiniMax M2 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-230b.md) — MiniMax, 2025-10-23
- [Grok 2.5 (270B)](https://llm-atlas-7or.pages.dev/models/grok-2-5-270b.md) — xAI, 2025-08-22
- [GPT-OSS (120B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-120b.md) — OpenAI, 2025-08-04
- [GPT-OSS (20B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-20b.md) — OpenAI, 2025-08-04
- [Qwen3 Coder Flash (30B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-coder-flash-30b-a3b.md) — Alibaba Cloud, 2025-07-31
- [GLM-4.5 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-355b.md) — Z.ai, 2025-07-28
- [GLM-4.5-Air (106B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-air.md) — Z.ai, 2025-07-28
- [SmolLM3 (3B)](https://llm-atlas-7or.pages.dev/models/smollm3-3b.md) — Hugging Face, 2025-06-19
- [Qwen3 (0.6B)](https://llm-atlas-7or.pages.dev/models/qwen3-0-6b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (235B-A22B)](https://llm-atlas-7or.pages.dev/models/qwen3-235b-a22b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (30B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-30b-a3b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (32B)](https://llm-atlas-7or.pages.dev/models/qwen3-32b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (4B)](https://llm-atlas-7or.pages.dev/models/qwen3-4b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (8B)](https://llm-atlas-7or.pages.dev/models/qwen3-8b.md) — Alibaba Cloud, 2025-04-28
- [Llama 4 Maverick (400B)](https://llm-atlas-7or.pages.dev/models/llama-4-maverick.md) — Meta, 2025-04-05
- [Mistral Small 3.1 (24B)](https://llm-atlas-7or.pages.dev/models/mistral-small-3-1-24b.md) — Mistral AI, 2025-03-18
- [Gemma 3 (27B)](https://llm-atlas-7or.pages.dev/models/gemma-3-27b.md) — Google, 2025-03-11
- [Phi-4 (14B)](https://llm-atlas-7or.pages.dev/models/phi-4.md) — Microsoft, 2024-12-12
- [Llama 3.2 (1B)](https://llm-atlas-7or.pages.dev/models/llama-3-2-1b.md) — Meta, 2024-09-25
- [Llama 3.2 (3B)](https://llm-atlas-7or.pages.dev/models/llama-3-2-3b.md) — Meta, 2024-09-25
- [Llama 3 (8B)](https://llm-atlas-7or.pages.dev/models/llama-3-8b.md) — Meta, 2024-04-18

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/gqa/
Web page: https://llm-atlas-7or.pages.dev/concepts/gqa/

## Concept: Multi-Head Latent Attention (MLA)

> Keys and values are compressed into a small latent vector that is cached instead of full K/V.

Category: Attention

Instead of sharing KV heads (as GQA does), MLA down-projects each token's keys and values into a compact latent and stores only that latent in the cache. At attention time it is projected back up.

Introduced with DeepSeek V2 and popularised by DeepSeek V3/R1, MLA reported quality on par with or better than MHA while caching far less per token. A decoupled RoPE path handles positional information.

Many large MoE models released since 2025 (Kimi, GLM-5, Mistral Large 3 and others) adopted MLA or variants of it.

**Trade-off:** Very small KV cache, at the price of extra projections and a more complex implementation.

## Models using MLA (28)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [GLM-5.2 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-2.md) — Z.ai, 2026-06-17
- [Kimi K2.7 Code (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-7-code.md) — Moonshot AI, 2026-06-12
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24
- [Ling 2.6 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-6-1t.md) — InclusionAI, 2026-04-23
- [Kimi K2.6 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-6.md) — Moonshot AI, 2026-04-20
- [GLM-5.1 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-1.md) — Z.ai, 2026-04-07
- [Mistral Small 4 (119B)](https://llm-atlas-7or.pages.dev/models/mistral-small-4.md) — Mistral AI, 2026-03-16
- [Sarvam (105B)](https://llm-atlas-7or.pages.dev/models/sarvam-105b.md) — Sarvam AI, 2026-03-03
- [Ling 2.5 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-5-1t.md) — InclusionAI, 2026-02-15
- [GLM-5 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-744b.md) — Z.ai, 2026-02-11
- [LongCat-Flash-Lite (68.5B-A3B)](https://llm-atlas-7or.pages.dev/models/longcat-flash-lite-68-5b-a3b.md) — Meituan, 2026-01-28
- [Kimi K2.5 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-5.md) — Moonshot AI, 2026-01-27
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [Mistral Large 3 (673B)](https://llm-atlas-7or.pages.dev/models/mistral-large-3.md) — Mistral AI, 2025-12-02
- [DeepSeek V3.2 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3-2.md) — DeepSeek, 2025-12-01
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30
- [Kimi K2 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2.md) — Moonshot AI, 2025-07-10
- [DeepSeek R1 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-r1.md) — DeepSeek, 2025-01-20
- [DeepSeek V3 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3.md) — DeepSeek, 2024-12-26

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/mla/
Web page: https://llm-atlas-7or.pages.dev/concepts/mla/

## Concept: Sliding Window Attention (SWA)

> Some layers only attend to a local window of recent tokens instead of the whole context.

Category: Attention

Local (sliding-window) layers restrict each token to the last N tokens, so their KV cache stops growing once the window is full. A few global layers are kept so information can still travel across the whole context.

Gemma 3 made the pattern famous with a 5:1 local-to-global ratio and a 1,024-token window. The ratio and window size are the main knobs, and it is very often paired with GQA.

**Trade-off:** Cheap long contexts; the risk is weaker long-range recall if too few global layers remain.

## Models using SWA (38)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Naive-N0.5-Flash (309B)](https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b.md) — NaiveAI, 2026-09-27
- [Xiaomi MiMo-V2.6-Flash-RL (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b.md) — Xiaomi, 2026-09-21
- [Xiaomi MiMo-V2.6-Pro-RL (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t.md) — Xiaomi, 2026-09-21
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [DeepSeek V4.1-Flash (552B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash.md) — DeepSeek, 2026-09-10
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [North Mini Code (30B-A3B)](https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b.md) — Cohere, 2026-06-05
- [Gemma 4 (12B)](https://llm-atlas-7or.pages.dev/models/gemma-4-12b.md) — Google, 2026-06-03
- [JetBrains Mellum2 Thinking (12B-A2.5B)](https://llm-atlas-7or.pages.dev/models/jetbrains-mellum2-thinking-12b-a2-5b.md) — JetBrains, 2026-06-01
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28
- [Xiaomi MiMo-V2.5 (310B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-310b.md) — Xiaomi, 2026-04-22
- [Xiaomi MiMo-V2.5-Pro (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-pro-1-02t.md) — Xiaomi, 2026-04-22
- [Gemma 4 (31B)](https://llm-atlas-7or.pages.dev/models/gemma-4-31b.md) — Google, 2026-04-02
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md) — Google, 2026-04-02
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02
- [Tiny Aya (3.35B)](https://llm-atlas-7or.pages.dev/models/tiny-aya-3-35b.md) — Cohere, 2026-02-13
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [Step 3.5 Flash (196B)](https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b.md) — StepFun, 2026-02-01
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [Xiaomi MiMo-V2-Flash (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-flash-309b.md) — Xiaomi, 2025-12-16
- [OLMo 3 (32B)](https://llm-atlas-7or.pages.dev/models/olmo-3-32b.md) — Ai2, 2025-11-20
- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md) — Ai2, 2025-11-20
- [Gemma 3 (270M)](https://llm-atlas-7or.pages.dev/models/gemma-3-270m.md) — Google, 2025-08-14
- [GPT-OSS (120B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-120b.md) — OpenAI, 2025-08-04
- [GPT-OSS (20B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-20b.md) — OpenAI, 2025-08-04
- [Mistral Small 3.1 (24B)](https://llm-atlas-7or.pages.dev/models/mistral-small-3-1-24b.md) — Mistral AI, 2025-03-18
- [Gemma 3 (27B)](https://llm-atlas-7or.pages.dev/models/gemma-3-27b.md) — Google, 2025-03-11

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/swa/
Web page: https://llm-atlas-7or.pages.dev/concepts/swa/

## Concept: Layer-wise attention budgeting (Attention budgeting)

> Different layers get different attention budgets: head counts, windows or full context.

Category: Attention

Rather than giving every layer an identical attention setup, the model spends more attention capacity in some layers and less in others.

Laguna XS.2 is the reference example: most of its 40 layers use a 512-token sliding window, while a minority keep full attention, and the head counts differ between the two kinds of layer.

**Trade-off:** Puts expensive global attention only where it earns its keep, cutting KV cache and compute.

## Models using Attention budgeting (3)

- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/attention-budgeting/
Web page: https://llm-atlas-7or.pages.dev/concepts/attention-budgeting/

## Concept: Gated Attention (Gated attention)

> A learned sigmoid gate scales the attention output before it re-enters the residual stream.

Category: Attention

After the softmax attention is computed, a gate derived from the layer input multiplies the result element-wise. The gate can damp heads or channels that are not useful for the current token.

Qwen3-Next and Qwen3.5 use it in their full-attention layers next to linear-attention layers; Trinity Large uses it in an attention-only stack. Reported benefits include better training stability and fewer attention-sink artefacts.

**Trade-off:** A cheap extra projection that adds control and stability to the attention output.

## Models using Gated attention (21)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md) — Alibaba Cloud, 2026-08-14
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md) — Upstage, 2026-07-22
- [BTL-3 (27B)](https://llm-atlas-7or.pages.dev/models/btl-3-27b.md) — Bad Theory Labs, 2026-07-20
- [Qwen3.6 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-27b.md) — Alibaba Cloud, 2026-04-22
- [Qwen3.6 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-35b-a3b.md) — Alibaba Cloud, 2026-04-15
- [Qwen3.5 (397B)](https://llm-atlas-7or.pages.dev/models/qwen3-5-397b.md) — Alibaba Cloud, 2026-02-16
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [Qwen3 Next (80B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-next-80b-a3b.md) — Alibaba Cloud, 2025-09-09

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/gated-attention/
Web page: https://llm-atlas-7or.pages.dev/concepts/gated-attention/

## Concept: Hybrid Attention (Hybrid attention)

> Mixes cheap linear-attention or state-space layers with a few full-attention layers.

Category: Attention

Linear attention variants (Gated DeltaNet, Kimi Delta Attention, Lightning Attention) and state-space layers such as Mamba-2 keep a fixed-size recurrent state instead of a per-token KV cache, so their memory does not grow with context.

Because they are weaker at precise retrieval, hybrids interleave them with a smaller number of full-attention layers, for example three linear layers per full-attention layer in the Qwen pattern.

**Trade-off:** Much lower long-context memory and compute, trading away some exact recall in the linear layers.

## Models using Hybrid attention (24)

- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md) — NVIDIA, 2026-10-15
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md) — Alibaba Cloud, 2026-08-14
- [Nemotron 3.5 Lightning (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-lightning-30b-a3b.md) — NVIDIA, 2026-08-11
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md) — Upstage, 2026-07-22
- [BTL-3 (27B)](https://llm-atlas-7or.pages.dev/models/btl-3-27b.md) — Bad Theory Labs, 2026-07-20
- [Soofi S (30B-A3B)](https://llm-atlas-7or.pages.dev/models/soofi-s-30b-a3b.md) — SOOFI Consortium, 2026-07-10
- [Nemotron 3 Ultra (550B-A55B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-ultra-550b-a55b.md) — NVIDIA, 2026-06-04
- [Ling 2.6 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-6-1t.md) — InclusionAI, 2026-04-23
- [Qwen3.6 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-27b.md) — Alibaba Cloud, 2026-04-22
- [Qwen3.6 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-35b-a3b.md) — Alibaba Cloud, 2026-04-15
- [Nemotron 3 Nano (4B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-4b.md) — NVIDIA, 2026-03-16
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md) — NVIDIA, 2026-03-11
- [Qwen3.5 (397B)](https://llm-atlas-7or.pages.dev/models/qwen3-5-397b.md) — Alibaba Cloud, 2026-02-16
- [Ling 2.5 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-5-1t.md) — InclusionAI, 2026-02-15
- [Nemotron 3 Nano (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-30b-a3b.md) — NVIDIA, 2025-12-04
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30
- [Qwen3 Next (80B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-next-80b-a3b.md) — Alibaba Cloud, 2025-09-09

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/hybrid-attention/
Web page: https://llm-atlas-7or.pages.dev/concepts/hybrid-attention/

## Concept: Short Convolution (ShortConv)

> A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.

Category: Attention

Each channel is convolved over a handful of recent positions with its own small kernel, and future positions are never seen. It is extremely cheap and gives layers a built-in notion of local order.

Kimi Linear applies ShortConv to the queries, keys and values in its delta-attention layers; Inkling places several ShortConv modules in every block. LFM-style models use convolutions as the main token mixer.

**Trade-off:** Near-free local mixing that complements linear attention, but it cannot see far by itself.

## Models using ShortConv (10)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md) — Upstage, 2026-07-22
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/shortconv/
Web page: https://llm-atlas-7or.pages.dev/concepts/shortconv/

## Concept: DeepSeek Sparse Attention (DSA)

> A lightweight indexer scores past tokens and each query attends only to its top-k picks.

Category: Sparse & compressed attention

DSA adds a small, fast indexer that estimates how relevant each earlier token is to the current query. A selector keeps the top-k tokens (2,048 in DeepSeek's code), and full attention is computed only over those.

Unlike a fixed sliding window, the selection is learned and can reach anywhere in the context. It was introduced with DeepSeek V3.2 on top of MLA and later adopted by GLM-5.

**Trade-off:** Long-context attention cost grows with k instead of the full sequence length.

## Models using DSA (10)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [Naive-N0.5-Flash (309B)](https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b.md) — NaiveAI, 2026-09-27
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [GLM-5.2 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-2.md) — Z.ai, 2026-06-17
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24
- [GLM-5.1 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-1.md) — Z.ai, 2026-04-07
- [GLM-5 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-744b.md) — Z.ai, 2026-02-11
- [DeepSeek V3.2 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3-2.md) — DeepSeek, 2025-12-01

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/deepseek-sparse-attention/
Web page: https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention/

## Concept: IndexShare (IndexShare)

> Nearby layers reuse the token selection computed by one DSA indexer.

Category: Sparse & compressed attention

Computing a DSA indexer in every layer still costs time. IndexShare computes the top-k selection once and lets the following layers (three, in GLM-5.2) reuse it.

The goal is to keep most of the quality of per-layer selection while cutting the indexer overhead.

**Trade-off:** Fewer indexer passes; the selection is slightly less tailored to each layer.

## Models using IndexShare (3)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.2 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-2.md) — Z.ai, 2026-06-17

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/indexshare/
Web page: https://llm-atlas-7or.pages.dev/concepts/indexshare/

## Concept: Qwen Sparse Attention (QSA)

> Like DSA, but the indexer picks whole four-token blocks rather than single tokens.

Category: Sparse & compressed attention

The indexer averages keys within each complete four-token block, scores the pooled keys against the query, and expands the winning blocks back into their tokens. The current incomplete block is always visible.

Selecting contiguous blocks is friendlier to memory access patterns than scattering over individual token positions.

**Trade-off:** Hardware-friendlier sparse reads at a coarser selection granularity.

## Models using QSA (1)

- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/qwen-sparse-attention/
Web page: https://llm-atlas-7or.pages.dev/concepts/qwen-sparse-attention/

## Concept: Compressed Sparse & Heavily Compressed Attention (CSA / HCA)

> Pool groups of older tokens into fewer KV entries, compressing along the sequence axis.

Category: Sparse & compressed attention

MLA shrinks each token's cache entry; CSA and HCA reduce how many entries there are. Groups of past tokens are pooled into compressed entries.

CSA compresses moderately and reads a sparse selection of entries; HCA compresses aggressively and reads all of them. DeepSeek V4 combines both to reach million-token contexts.

**Trade-off:** Drastically smaller long-context caches, with some loss of token-level detail in the far past.

## Models using CSA / HCA (2)

- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/csa-hca/
Web page: https://llm-atlas-7or.pages.dev/concepts/csa-hca/

## Concept: Compressed Convolutional Attention (CCA)

> Attention computed directly in a compressed latent space, with convolutional mixing.

Category: Sparse & compressed attention

Where MLA uses a latent mainly as a compact cache format and decompresses it for attention, CCA performs the attention operation itself in the compressed space and adds convolutions to mix information.

ZAYA1-8B by Zyphra uses CCA together with grouped-query attention.

**Trade-off:** Saves both cache memory and attention FLOPs; still a young technique.

## Models using CCA (1)

- [ZAYA1-8B (8.4B)](https://llm-atlas-7or.pages.dev/models/zaya1-8b.md) — Zyphra, 2026-05-06

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/compressed-convolutional-attention/
Web page: https://llm-atlas-7or.pages.dev/concepts/compressed-convolutional-attention/

## Concept: Mixture of Experts (MoE)

> Many feed-forward experts, but a router sends each token to only a few of them.

Category: Feed-forward & experts

The single feed-forward block is replaced by many expert blocks. A router picks a small number (often 8) per token, so the model has a huge total parameter count while each token only touches a fraction of them, the active parameters.

Many designs add a shared expert that every token uses, and keep the first few layers dense. DeepSeek V3 is the template much of the 2025–2026 wave followed.

**Trade-off:** More knowledge capacity per unit of compute, but all experts must still fit in memory.

## Models using MoE (79)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [MiniMax M3.1-Flash (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-1-flash-428b.md) — MiniMax, 2026-10-19
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md) — NVIDIA, 2026-10-15
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Naive-N0.5-Flash (309B)](https://llm-atlas-7or.pages.dev/models/naive-n0-5-flash-309b.md) — NaiveAI, 2026-09-27
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Xiaomi MiMo-V2.6-Flash-RL (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b.md) — Xiaomi, 2026-09-21
- [Xiaomi MiMo-V2.6-Pro-RL (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t.md) — Xiaomi, 2026-09-21
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [DeepSeek V4.1-Flash (552B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash.md) — DeepSeek, 2026-09-10
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Nemotron 3.5 Lightning (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-lightning-30b-a3b.md) — NVIDIA, 2026-08-11
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md) — Upstage, 2026-07-22
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Soofi S (30B-A3B)](https://llm-atlas-7or.pages.dev/models/soofi-s-30b-a3b.md) — SOOFI Consortium, 2026-07-10
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [GLM-5.2 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-2.md) — Z.ai, 2026-06-17
- [MiniMax M3 (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-428b.md) — MiniMax, 2026-06-13
- [Kimi K2.7 Code (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-7-code.md) — Moonshot AI, 2026-06-12
- [North Mini Code (30B-A3B)](https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b.md) — Cohere, 2026-06-05
- [Nemotron 3 Ultra (550B-A55B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-ultra-550b-a55b.md) — NVIDIA, 2026-06-04
- [JetBrains Mellum2 Thinking (12B-A2.5B)](https://llm-atlas-7or.pages.dev/models/jetbrains-mellum2-thinking-12b-a2-5b.md) — JetBrains, 2026-06-01
- [LFM2.5 (8B-A1B)](https://llm-atlas-7or.pages.dev/models/lfm2-5-8b-a1b.md) — Liquid AI, 2026-05-28
- [Command A+ (218B-A25B)](https://llm-atlas-7or.pages.dev/models/command-a-218b-a25b.md) — Cohere, 2026-05-20
- [ZAYA1-8B (8.4B)](https://llm-atlas-7or.pages.dev/models/zaya1-8b.md) — Zyphra, 2026-05-06
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24
- [Tencent Hy3-preview (295B-A21B)](https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b.md) — Tencent, 2026-04-23
- [Qwen3.6 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-27b.md) — Alibaba Cloud, 2026-04-22
- [Xiaomi MiMo-V2.5 (310B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-310b.md) — Xiaomi, 2026-04-22
- [Xiaomi MiMo-V2.5-Pro (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-pro-1-02t.md) — Xiaomi, 2026-04-22
- [Kimi K2.6 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-6.md) — Moonshot AI, 2026-04-20
- [Qwen3.6 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-6-35b-a3b.md) — Alibaba Cloud, 2026-04-15
- [GLM-5.1 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-1.md) — Z.ai, 2026-04-07
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md) — Google, 2026-04-02
- [MiniMax M2.7 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b.md) — MiniMax, 2026-03-18
- [Mistral Small 4 (119B)](https://llm-atlas-7or.pages.dev/models/mistral-small-4.md) — Mistral AI, 2026-03-16
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md) — NVIDIA, 2026-03-11
- [Sarvam (30B)](https://llm-atlas-7or.pages.dev/models/sarvam-30b.md) — Sarvam AI, 2026-03-03
- [Sarvam (105B)](https://llm-atlas-7or.pages.dev/models/sarvam-105b.md) — Sarvam AI, 2026-03-03
- [Qwen3.5 (397B)](https://llm-atlas-7or.pages.dev/models/qwen3-5-397b.md) — Alibaba Cloud, 2026-02-16
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [GLM-5 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-744b.md) — Z.ai, 2026-02-11
- [Step 3.5 Flash (196B)](https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b.md) — StepFun, 2026-02-01
- [LongCat-Flash-Lite (68.5B-A3B)](https://llm-atlas-7or.pages.dev/models/longcat-flash-lite-68-5b-a3b.md) — Meituan, 2026-01-28
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [Kimi K2.5 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2-5.md) — Moonshot AI, 2026-01-27
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [Xiaomi MiMo-V2-Flash (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-flash-309b.md) — Xiaomi, 2025-12-16
- [Nemotron 3 Nano (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-nano-30b-a3b.md) — NVIDIA, 2025-12-04
- [Mistral Large 3 (673B)](https://llm-atlas-7or.pages.dev/models/mistral-large-3.md) — Mistral AI, 2025-12-02
- [DeepSeek V3.2 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3-2.md) — DeepSeek, 2025-12-01
- [INTELLECT-3 (106B)](https://llm-atlas-7or.pages.dev/models/intellect-3.md) — Prime Intellect, 2025-11-26
- [MiniMax M2 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-230b.md) — MiniMax, 2025-10-23
- [Qwen3 Next (80B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-next-80b-a3b.md) — Alibaba Cloud, 2025-09-09
- [Grok 2.5 (270B)](https://llm-atlas-7or.pages.dev/models/grok-2-5-270b.md) — xAI, 2025-08-22
- [GPT-OSS (120B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-120b.md) — OpenAI, 2025-08-04
- [GPT-OSS (20B)](https://llm-atlas-7or.pages.dev/models/gpt-oss-20b.md) — OpenAI, 2025-08-04
- [Qwen3 Coder Flash (30B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-coder-flash-30b-a3b.md) — Alibaba Cloud, 2025-07-31
- [GLM-4.5 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-355b.md) — Z.ai, 2025-07-28
- [GLM-4.5-Air (106B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-air.md) — Z.ai, 2025-07-28
- [Kimi K2 (1T)](https://llm-atlas-7or.pages.dev/models/kimi-k2.md) — Moonshot AI, 2025-07-10
- [Qwen3 (235B-A22B)](https://llm-atlas-7or.pages.dev/models/qwen3-235b-a22b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (30B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-30b-a3b.md) — Alibaba Cloud, 2025-04-28
- [Llama 4 Maverick (400B)](https://llm-atlas-7or.pages.dev/models/llama-4-maverick.md) — Meta, 2025-04-05
- [DeepSeek R1 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-r1.md) — DeepSeek, 2025-01-20
- [DeepSeek V3 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3.md) — DeepSeek, 2024-12-26

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/moe/
Web page: https://llm-atlas-7or.pages.dev/concepts/moe/

## Concept: Latent MoE (Latent MoE)

> Routed experts operate in a smaller latent space instead of the full hidden width.

Category: Feed-forward & experts

Tokens are projected down before they enter the routed experts and projected back up afterwards. Experts become cheaper, so a model can afford more of them or activate more per token.

Nemotron 3 Super and Ultra, Kimi K3 and Muse Spark Open use this layout with different compression ratios and normalization details.

**Trade-off:** Cheaper experts and communication; the bottleneck must be sized carefully.

## Models using Latent MoE (5)

- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md) — NVIDIA, 2026-10-15
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md) — NVIDIA, 2026-03-11

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/latent-moe/
Web page: https://llm-atlas-7or.pages.dev/concepts/latent-moe/

## Concept: PolyNorm (PolyNorm)

> A learned activation mixing normalised linear, quadratic and cubic terms.

Category: Feed-forward & experts

Feed-forward blocks moved from GELU to SwiGLU-style gating with a fixed SiLU. PolyNorm replaces that fixed function with a learned combination of separately RMS-normalised powers of the pre-activation.

Motif 3 Beta uses a grouped PolyNorm inside each expert.

**Trade-off:** A more flexible nonlinearity for a little extra compute and tuning.

## Models using PolyNorm (1)

- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/polynorm/
Web page: https://llm-atlas-7or.pages.dev/concepts/polynorm/

## Concept: Manifold-constrained Hyper-Connections (mHC)

> Several parallel residual streams with learned, constrained mixing between them.

Category: Residual stream

The single residual stream is widened into several streams (four in DeepSeek V4). Learned mappings decide how sublayer inputs are read from, and outputs written back to, these streams.

Unconstrained hyper-connections can amplify signals and destabilise training; mHC constrains the mixing matrix to a well-behaved manifold so information is mixed without blowing up. DeepSeek V4.1 introduced a cheaper single-pass variant.

**Trade-off:** A richer residual path for a modest training overhead.

## Models using mHC (6)

- [DeepSeek V4.1-Flash (552B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash.md) — DeepSeek, 2026-09-10
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/mhc/
Web page: https://llm-atlas-7or.pages.dev/concepts/mhc/

## Concept: Gated Residuals (GR)

> Gated read/write modules over four residual streams, without mHC's mixing matrix.

Category: Residual stream

Before a sublayer, a GR Read module normalises each of four residual streams, gates them element-wise and averages them into a normal-width input. Afterwards, GR Write adds the output back to all four streams with learned gates.

It was introduced with Qwen3.8-Flash-Next and is related in spirit to mHC.

**Trade-off:** Wider residual capacity with simple gating instead of a constrained mixing matrix.

## Models using GR (1)

- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/gated-residuals/
Web page: https://llm-atlas-7or.pages.dev/concepts/gated-residuals/

## Concept: Attention Residuals (AttnRes)

> Learned, attention-style weights over earlier layer outputs replace the plain residual sum.

Category: Residual stream

A standard residual connection adds every earlier update with weight one. AttnRes lets each layer attend over earlier layer outputs and weight their contributions.

Full and block variants exist; Kimi Linear already used the idea. Reported gains are consistent but modest, for a few percent extra training and inference cost.

**Trade-off:** Smarter depth-wise information flow for a small overhead.

## Models using AttnRes (3)

- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/attention-residuals/
Web page: https://llm-atlas-7or.pages.dev/concepts/attention-residuals/

## Concept: Looped Transformer (Looped depth)

> The same blocks are applied several times, reusing weights across depth.

Category: Residual stream

With L distinct blocks applied T times, the model gets L × T effective layers for the parameter cost of L. It is depth-wise weight tying, sometimes called recurrent depth.

Nanbeige 4.2 uses two fixed passes, Ouro uses four passes with a learned exit gate, and Dory1.0 loops a shared middle stack.

**Trade-off:** More compute per parameter; extra passes add latency and, depending on the design, KV cache.

## Models using Looped depth (3)

- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Nanbeige 4.2 (3B)](https://llm-atlas-7or.pages.dev/models/nanbeige-4-2-3b.md) — BOSS Zhipin, 2026-07-21
- [Ouro-Thinking (2.6B)](https://llm-atlas-7or.pages.dev/models/ouro-thinking-2-6b.md) — ByteDance, 2025-10-29

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing/
Web page: https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing/

## Concept: QK-Norm (QK-Norm)

> RMSNorm applied to queries and keys before attention to keep logits well-behaved.

Category: Positions & normalization

Normalising queries and keys (usually before RoPE) prevents attention logits from growing too large, a common source of loss spikes in large-scale training.

OLMo 2 and Qwen3 helped make it standard; it is now a default ingredient in many recipes.

**Trade-off:** A stabiliser rather than an efficiency trick, and it costs almost nothing.

## Models using QK-Norm (40)

- [LFM3 (1.7B)](https://llm-atlas-7or.pages.dev/models/lfm3-1-7b.md) — Liquid AI, 2026-10-27
- [MiniMax M3.1-Flash (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-1-flash-428b.md) — MiniMax, 2026-10-19
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Dory1.0 (4B)](https://llm-atlas-7or.pages.dev/models/dory1-0-4b.md) — NVIDIA, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md) — Alibaba Cloud, 2026-08-14
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Laguna S 2.1 (118B)](https://llm-atlas-7or.pages.dev/models/laguna-s-2-1.md) — Poolside, 2026-07-21
- [BTL-3 (27B)](https://llm-atlas-7or.pages.dev/models/btl-3-27b.md) — Bad Theory Labs, 2026-07-20
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [Laguna XS 2.1 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2-1.md) — Poolside, 2026-07-02
- [MiniMax M3 (428B)](https://llm-atlas-7or.pages.dev/models/minimax-m3-428b.md) — MiniMax, 2026-06-13
- [Gemma 4 (12B)](https://llm-atlas-7or.pages.dev/models/gemma-4-12b.md) — Google, 2026-06-03
- [Laguna XS.2 (33B)](https://llm-atlas-7or.pages.dev/models/laguna-xs-2.md) — Poolside, 2026-04-28
- [Tencent Hy3-preview (295B-A21B)](https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b.md) — Tencent, 2026-04-23
- [Gemma 4 (31B)](https://llm-atlas-7or.pages.dev/models/gemma-4-31b.md) — Google, 2026-04-02
- [Gemma 4 (26B-A4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-26b-a4b.md) — Google, 2026-04-02
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02
- [MiniMax M2.7 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b.md) — MiniMax, 2026-03-18
- [Sarvam (30B)](https://llm-atlas-7or.pages.dev/models/sarvam-30b.md) — Sarvam AI, 2026-03-03
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [OLMo 3 (32B)](https://llm-atlas-7or.pages.dev/models/olmo-3-32b.md) — Ai2, 2025-11-20
- [OLMo 3 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-3-7b.md) — Ai2, 2025-11-20
- [MiniMax M2 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-230b.md) — MiniMax, 2025-10-23
- [Gemma 3 (270M)](https://llm-atlas-7or.pages.dev/models/gemma-3-270m.md) — Google, 2025-08-14
- [GLM-4.5 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-355b.md) — Z.ai, 2025-07-28
- [Qwen3 (235B-A22B)](https://llm-atlas-7or.pages.dev/models/qwen3-235b-a22b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (32B)](https://llm-atlas-7or.pages.dev/models/qwen3-32b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (4B)](https://llm-atlas-7or.pages.dev/models/qwen3-4b.md) — Alibaba Cloud, 2025-04-28
- [Qwen3 (8B)](https://llm-atlas-7or.pages.dev/models/qwen3-8b.md) — Alibaba Cloud, 2025-04-28
- [Gemma 3 (27B)](https://llm-atlas-7or.pages.dev/models/gemma-3-27b.md) — Google, 2025-03-11
- [OLMo 2 (7B)](https://llm-atlas-7or.pages.dev/models/olmo-2-7b.md) — Ai2, 2024-11-25

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/qk-norm/
Web page: https://llm-atlas-7or.pages.dev/concepts/qk-norm/

## Concept: No Positional Embeddings (NoPE)

> Some or all layers drop explicit positional encodings and rely on the causal mask.

Category: Positions & normalization

Because a decoder's causal mask already prevents tokens from seeing the future, the model can infer order implicitly. NoPE layers omit RoPE entirely.

Studies suggest this can help length generalisation. SmolLM3 removes RoPE from every fourth layer; Kimi K3 goes NoPE throughout.

**Trade-off:** Potentially better extrapolation to longer contexts; less explicit positional signal.

## Models using NoPE (18)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [Recursive Flint (40B-A2.5B)](https://llm-atlas-7or.pages.dev/models/recursive-flint-40b-a2-5b.md) — Recursive, 2026-10-18
- [Reflection Beam (501B)](https://llm-atlas-7or.pages.dev/models/reflection-beam-501b.md) — Reflection, 2026-10-16
- [Muse Spark Open (816B-A42B)](https://llm-atlas-7or.pages.dev/models/muse-spark-open-816b-a42b.md) — Meta, 2026-10-15
- [Kolibri-1 (78B)](https://llm-atlas-7or.pages.dev/models/kolibri-1-78b.md) — Aleph Alpha, 2026-10-03
- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Muse Glimmer (30B)](https://llm-atlas-7or.pages.dev/models/muse-glimmer-30b.md) — Meta, 2026-08-10
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Solar Open 2 (250B)](https://llm-atlas-7or.pages.dev/models/solar-open-2.md) — Upstage, 2026-07-22
- [Soofi S (30B-A3B)](https://llm-atlas-7or.pages.dev/models/soofi-s-30b-a3b.md) — SOOFI Consortium, 2026-07-10
- [North Mini Code (30B-A3B)](https://llm-atlas-7or.pages.dev/models/north-mini-code-30b-a3b.md) — Cohere, 2026-06-05
- [Command A+ (218B-A25B)](https://llm-atlas-7or.pages.dev/models/command-a-218b-a25b.md) — Cohere, 2026-05-20
- [Sarvam (105B)](https://llm-atlas-7or.pages.dev/models/sarvam-105b.md) — Sarvam AI, 2026-03-03
- [Tiny Aya (3.35B)](https://llm-atlas-7or.pages.dev/models/tiny-aya-3-35b.md) — Cohere, 2026-02-13
- [Arcee AI Trinity Large (400B)](https://llm-atlas-7or.pages.dev/models/arcee-ai-trinity-large-400b.md) — Arcee AI, 2026-01-27
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30
- [SmolLM3 (3B)](https://llm-atlas-7or.pages.dev/models/smollm3-3b.md) — Hugging Face, 2025-06-19

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/nope/
Web page: https://llm-atlas-7or.pages.dev/concepts/nope/

## Concept: Cross-Layer KV Sharing (KV sharing)

> Later layers reuse key/value states produced by earlier layers.

Category: Efficiency & sharing

GQA shares KV heads across query heads within a layer; cross-layer sharing goes further and lets whole layers reuse another layer's keys and values, so fewer layers add to the cache.

Gemma 4 E2B and E4B are prominent examples of the idea from cross-layer attention research.

**Trade-off:** Smaller cache and less compute for long contexts, with less per-layer flexibility.

## Models using KV sharing (4)

- [Mistral Large 4 (1T)](https://llm-atlas-7or.pages.dev/models/mistral-large-4.md) — Mistral AI, 2026-10-29
- [DeepSeek V4.1-Flash (552B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-1-flash.md) — DeepSeek, 2026-09-10
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/kv-sharing/
Web page: https://llm-atlas-7or.pages.dev/concepts/kv-sharing/

## Concept: Per-Layer Embeddings (PLE)

> Each layer gets its own token-specific embedding signal from a large lookup table.

Category: Efficiency & sharing

Lookup tables are cheap to run and can live outside fast memory, so small models can store extra token-specific information per layer without making the transformer stack itself more expensive.

It is why Gemma 4 E2B is described as 2.3B effective parameters but 5.1B including embeddings. The “E” stands for effective.

**Trade-off:** More capacity for small on-device models at very low compute cost.

## Models using PLE (3)

- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Gemma 4 (E2B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e2b.md) — Google, 2026-04-02
- [Gemma 4 (E4B)](https://llm-atlas-7or.pages.dev/models/gemma-4-e4b.md) — Google, 2026-04-02

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/per-layer-embeddings/
Web page: https://llm-atlas-7or.pages.dev/concepts/per-layer-embeddings/

## Concept: Multi-Token Prediction (MTP)

> Extra heads learn to predict several future tokens, and can draft tokens at inference.

Category: Training objectives

Besides the next token, small additional modules predict tokens further ahead. DeepSeek's version keeps a causal chain where each module conditions on the previous prediction.

The extra targets give a denser training signal, and at inference the MTP modules act as a built-in draft model for speculative decoding.

**Trade-off:** Denser training signal and faster decoding, for some extra parameters.

## Models using MTP (34)

- [Nemotron 3.5 Super VL (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-super-vl-120b-a12b.md) — NVIDIA, 2026-10-15
- [Ling 3.1 Flash (560B)](https://llm-atlas-7or.pages.dev/models/ling-3-1-flash.md) — InclusionAI, 2026-10-13
- [Xiaomi MiMo-V2.6-Flash-RL (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-flash-rl-309b.md) — Xiaomi, 2026-09-21
- [Xiaomi MiMo-V2.6-Pro-RL (1.02T)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-6-pro-rl-1-02t.md) — Xiaomi, 2026-09-21
- [Step-5 Preview (600B)](https://llm-atlas-7or.pages.dev/models/step-5-preview-600b.md) — StepFun, 2026-09-20
- [Tencent Hy4-preview (770B-A49B)](https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b.md) — Tencent, 2026-08-28
- [GLM-5.3-Flash (320B)](https://llm-atlas-7or.pages.dev/models/glm-5-3-flash.md) — Z.ai, 2026-08-26
- [Qwen3.8-Flash-Next (125B-A6B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-flash-next-125b-a6b.md) — Alibaba Cloud, 2026-08-26
- [Ornith 1.5 (35B-A3B)](https://llm-atlas-7or.pages.dev/models/ornith-1-5-35b-a3b.md) — Ornith, 2026-08-23
- [Qwen3.8 (27B)](https://llm-atlas-7or.pages.dev/models/qwen3-8-27b.md) — Alibaba Cloud, 2026-08-14
- [Nemotron 3.5 Lightning (30B-A3B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-5-lightning-30b-a3b.md) — NVIDIA, 2026-08-11
- [Ling 3.0 Flash (124B)](https://llm-atlas-7or.pages.dev/models/ling-3-0-flash.md) — InclusionAI, 2026-08-04
- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21
- [Inkling (975B)](https://llm-atlas-7or.pages.dev/models/inkling.md) — Thinking Machines Lab, 2026-07-15
- [DeepSeek V4-Flash (284B)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-flash.md) — DeepSeek, 2026-04-24
- [DeepSeek V4-Pro (1.6T)](https://llm-atlas-7or.pages.dev/models/deepseek-v4-pro.md) — DeepSeek, 2026-04-24
- [Ling 2.6 (1T)](https://llm-atlas-7or.pages.dev/models/ling-2-6-1t.md) — InclusionAI, 2026-04-23
- [Tencent Hy3-preview (295B-A21B)](https://llm-atlas-7or.pages.dev/models/tencent-hy3-preview-295b-a21b.md) — Tencent, 2026-04-23
- [Xiaomi MiMo-V2.5 (310B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-5-310b.md) — Xiaomi, 2026-04-22
- [GLM-5.1 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-1.md) — Z.ai, 2026-04-07
- [MiniMax M2.7 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-7-230b.md) — MiniMax, 2026-03-18
- [Nemotron 3 Super (120B-A12B)](https://llm-atlas-7or.pages.dev/models/nemotron-3-super-120b-a12b.md) — NVIDIA, 2026-03-11
- [MiniMax-M2.5 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-5-230b.md) — MiniMax, 2026-02-12
- [GLM-5 (744B)](https://llm-atlas-7or.pages.dev/models/glm-5-744b.md) — Z.ai, 2026-02-11
- [Step 3.5 Flash (196B)](https://llm-atlas-7or.pages.dev/models/step-3-5-flash-196b.md) — StepFun, 2026-02-01
- [GLM-4.7 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-7-355b.md) — Z.ai, 2025-12-22
- [Xiaomi MiMo-V2-Flash (309B)](https://llm-atlas-7or.pages.dev/models/xiaomi-mimo-v2-flash-309b.md) — Xiaomi, 2025-12-16
- [DeepSeek V3.2 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3-2.md) — DeepSeek, 2025-12-01
- [MiniMax M2 (230B)](https://llm-atlas-7or.pages.dev/models/minimax-m2-230b.md) — MiniMax, 2025-10-23
- [Qwen3 Next (80B-A3B)](https://llm-atlas-7or.pages.dev/models/qwen3-next-80b-a3b.md) — Alibaba Cloud, 2025-09-09
- [GLM-4.5 (355B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-355b.md) — Z.ai, 2025-07-28
- [GLM-4.5-Air (106B)](https://llm-atlas-7or.pages.dev/models/glm-4-5-air.md) — Z.ai, 2025-07-28
- [DeepSeek R1 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-r1.md) — DeepSeek, 2025-01-20
- [DeepSeek V3 (671B)](https://llm-atlas-7or.pages.dev/models/deepseek-v3.md) — DeepSeek, 2024-12-26

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/mtp/
Web page: https://llm-atlas-7or.pages.dev/concepts/mtp/