Architecture concepts

• MHA4 models

  Multi-Head Attention

  The original transformer attention: every head has its own queries, keys and values.

  First here: GPT-2 XL 1.5B · Nov 5, 2019

• GQA78 models

  Grouped-Query Attention

  Several query heads share one key/value head, shrinking the KV cache with little quality loss.

  First here: Llama 3 8B · Apr 18, 2024

• MLA28 models

  Multi-Head Latent Attention

  Keys and values are compressed into a small latent vector that is cached instead of full K/V.

  First here: DeepSeek V3 · Dec 26, 2024

• SWA38 models

  Sliding Window Attention

  Some layers only attend to a local window of recent tokens instead of the whole context.

  First here: Gemma 3 27B · Mar 11, 2025

• Attention budgeting3 models

  Layer-wise attention budgeting

  Different layers get different attention budgets: head counts, windows or full context.

  First here: Laguna XS.2 · Apr 28, 2026

• Gated attention21 models

  Gated Attention

  A learned sigmoid gate scales the attention output before it re-enters the residual stream.

  First here: Qwen3 Next 80B-A3B · Sep 9, 2025

• Hybrid attention24 models

  Hybrid Attention

  Mixes cheap linear-attention or state-space layers with a few full-attention layers.

  First here: Qwen3 Next 80B-A3B · Sep 9, 2025

• ShortConv10 models

  Short Convolution

  A tiny causal depthwise convolution (kernel ≈ 4) that mixes each channel over the last few tokens.

  First here: Kimi Linear 48B-A3B · Oct 30, 2025

• DSA10 models

  DeepSeek Sparse Attention

  A lightweight indexer scores past tokens and each query attends only to its top-k picks.

  First here: DeepSeek V3.2 · Dec 1, 2025

• IndexShare3 models

  IndexShare

  Nearby layers reuse the token selection computed by one DSA indexer.

  First here: GLM-5.2 · Jun 17, 2026

• QSA1 models

  Qwen Sparse Attention

  Like DSA, but the indexer picks whole four-token blocks rather than single tokens.

  First here: Qwen3.8-Flash-Next 125B-A6B · Aug 26, 2026

• CSA / HCA2 models

  Compressed Sparse & Heavily Compressed Attention

  Pool groups of older tokens into fewer KV entries, compressing along the sequence axis.

  First here: DeepSeek V4-Pro · Apr 24, 2026

• CCA1 models

  Compressed Convolutional Attention

  Attention computed directly in a compressed latent space, with convolutional mixing.

  First here: ZAYA1-8B · May 6, 2026

• MoE79 models

  Mixture of Experts

  Many feed-forward experts, but a router sends each token to only a few of them.

  First here: DeepSeek V3 · Dec 26, 2024

• Latent MoE5 models

  Latent MoE

  Routed experts operate in a smaller latent space instead of the full hidden width.

  First here: Nemotron 3 Super 120B-A12B · Mar 11, 2026

• PolyNorm1 models

  PolyNorm

  A learned activation mixing normalised linear, quadratic and cubic terms.

  First here: Motif 3 Beta · Jul 21, 2026

• mHC6 models

  Manifold-constrained Hyper-Connections

  Several parallel residual streams with learned, constrained mixing between them.

  First here: DeepSeek V4-Pro · Apr 24, 2026

• GR1 models

  Gated Residuals

  Gated read/write modules over four residual streams, without mHC's mixing matrix.

  First here: Qwen3.8-Flash-Next 125B-A6B · Aug 26, 2026

• AttnRes3 models

  Attention Residuals

  Learned, attention-style weights over earlier layer outputs replace the plain residual sum.

  First here: Kimi Linear 48B-A3B · Oct 30, 2025

• Looped depth3 models

  Looped Transformer

  The same blocks are applied several times, reusing weights across depth.

  First here: Ouro-Thinking 2.6B · Oct 29, 2025

• QK-Norm40 models

  QK-Norm

  RMSNorm applied to queries and keys before attention to keep logits well-behaved.

  First here: OLMo 2 7B · Nov 25, 2024

• NoPE18 models

  No Positional Embeddings

  Some or all layers drop explicit positional encodings and rely on the causal mask.

  First here: SmolLM3 3B · Jun 19, 2025

• KV sharing4 models

  Cross-Layer KV Sharing

  Later layers reuse key/value states produced by earlier layers.

  First here: Gemma 4 E4B · Apr 2, 2026

• PLE3 models

  Per-Layer Embeddings

  Each layer gets its own token-specific embedding signal from a large lookup table.

  First here: Gemma 4 E4B · Apr 2, 2026

• MTP34 models

  Multi-Token Prediction

  Extra heads learn to predict several future tokens, and can draft tokens at inference.

  First here: DeepSeek V3 · Dec 26, 2024

Reference pages on the source gallery

• KV cache / token (bf16) How the per-token KV cache figures on every card are estimated.
• LLM memory calculator Estimate weight and KV-cache memory by model, context length, batch size and precision.
• Artificial Analysis Intelligence Index How the AA Intelligence Index scores on the cards are read.
• Gallery changelog Dated public updates to the source gallery.