Skip to content

118 models from 44 labs

All architectures

Filter by decoder type, concept or lab. Filters are saved in the URL, so any view can be shared.
118 models
  • New
    Architecture diagram of Mistral Large 4 (1T)

    Mistral Large 4 (1T)

    Mistral AI, released 2026-10-29. Sparse MoE decoder, 1T total, 52B active, 1M context. Attention: MLA with modified DeepSeek Sparse Attention, index reuse, and KV sharing.

  • New
    Architecture diagram of LFM3 (1.7B)

    LFM3 (1.7B)

    Liquid AI, released 2026-10-27. Dense hybrid decoder, 1.7B parameters, 256K context. Attention: ShortConv plus gated GQA with QK-Norm and sliding-window/global attention.

  • New
    Architecture diagram of MiniMax M3.1-Flash (428B)

    MiniMax M3.1-Flash (428B)

    MiniMax, released 2026-10-19. Sparse MoE decoder, 428B total, 23B active, 1M context. Attention: GQA with QK-Norm and MiniMax Sparse Attention.

  • New
    Architecture diagram of Recursive Flint (40B-A2.5B)

    Recursive Flint (40B-A2.5B)

    Recursive, released 2026-10-18. Sparse MoE decoder, 40B total, 2.5B active, 256K context. Attention: GQA with QK-Norm, learned attention sinks, and sliding-window/global attention.

  • New
    Architecture diagram of Reflection Beam (501B)

    Reflection Beam (501B)

    Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.

  • New
    Architecture diagram of Dory1.0 (4B)

    Dory1.0 (4B)

    NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.

  • Architecture diagram of Muse Spark Open (816B-A42B)

    Muse Spark Open (816B-A42B)

    Meta, released 2026-10-15. Sparse MoE decoder, 816B total, 42B active, 1M context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.

  • Architecture diagram of Nemotron 3.5 Super VL (120B-A12B)

    Nemotron 3.5 Super VL (120B-A12B)

    NVIDIA, released 2026-10-15. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with eight GQA layers.

  • Architecture diagram of Ling 3.1 Flash (560B)

    Ling 3.1 Flash (560B)

    InclusionAI, released 2026-10-13. Hybrid MoE decoder, 560B total, 25B active, 1M context. Attention: 7:1 Kimi Delta Attention and gated MLA.

  • Architecture diagram of Kolibri-1 (78B)

    Kolibri-1 (78B)

    Aleph Alpha, released 2026-10-03. Sparse MoE decoder, 78B total, 3.5B active, 256K context. Attention: GQA with QK-Norm and 4:1 sliding-window/global attention.

  • Architecture diagram of Naive-N0.5-Flash (309B)

    Naive-N0.5-Flash (309B)

    NaiveAI, released 2026-09-27. Sparse MoE decoder, 309B total, 16B active, 1M context. Attention: GQA with sliding-window attention and DeepSeek Sparse Attention.

  • Architecture diagram of Ember-1 (2.8T)

    Ember-1 (2.8T)

    Fireworks AI, released 2026-09-23. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.

  • Architecture diagram of Xiaomi MiMo-V2.6-Flash-RL (309B)

    Xiaomi MiMo-V2.6-Flash-RL (309B)

    Xiaomi, released 2026-09-21. Sparse MoE decoder, 309B total, 15B active, 1M context. Attention: GQA with sliding-window/global attention.

  • Architecture diagram of Xiaomi MiMo-V2.6-Pro-RL (1.02T)

    Xiaomi MiMo-V2.6-Pro-RL (1.02T)

    Xiaomi, released 2026-09-21. Sparse MoE decoder, 1.02T total, 42B active, 1M context. Attention: GQA with 6:1 sliding-window/global attention.

  • Architecture diagram of Step-5 Preview (600B)

    Step-5 Preview (600B)

    StepFun, released 2026-09-20. Sparse MoE decoder, 600B total, 27B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention.

  • Architecture diagram of DeepSeek V4.1-Flash (552B)

    DeepSeek V4.1-Flash (552B)

    DeepSeek, released 2026-09-10. Sparse MoE decoder, 552B total, 8B active, 1M context. Attention: MLA-style CSA2 + SWA with cross-layer KV and sparse-index reuse.

  • Architecture diagram of Tencent Hy4-preview (770B-A49B)

    Tencent Hy4-preview (770B-A49B)

    Tencent, released 2026-08-28. Sparse MoE decoder, 770B total, 49B active, 1M context. Attention: Gated MLA with DeepSeek Sparse Attention and IndexCache.

  • Architecture diagram of GLM-5.3-Flash (320B)

    GLM-5.3-Flash (320B)

    Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.

  • Architecture diagram of Qwen3.8-Flash-Next (125B-A6B)

    Qwen3.8-Flash-Next (125B-A6B)

    Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).

  • Architecture diagram of Ornith 1.5 (35B-A3B)

    Ornith 1.5 (35B-A3B)

    Ornith, released 2026-08-23. Hybrid MoE decoder, 35B total, 3B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of Qwen3.8 (27B)

    Qwen3.8 (27B)

    Alibaba Cloud, released 2026-08-14. Dense hybrid decoder, 27B parameters, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of Nemotron 3.5 Lightning (30B-A3B)

    Nemotron 3.5 Lightning (30B-A3B)

    NVIDIA, released 2026-08-11. Hybrid MoE decoder, 30B total, 3B active, 1M context. Attention: Mostly Mamba-2 with six GQA layers.

  • Architecture diagram of Muse Glimmer (30B)

    Muse Glimmer (30B)

    Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.

  • Architecture diagram of Ling 3.0 Flash (124B)

    Ling 3.0 Flash (124B)

    InclusionAI, released 2026-08-04. Hybrid MoE decoder, 124B total, 5.1B active, 256K context. Attention: 5:1 Kimi Delta Attention and gated MLA.

  • Architecture diagram of Kimi K3 (2.8T)

    Kimi K3 (2.8T)

    Moonshot AI, released 2026-07-27. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.

  • Architecture diagram of Solar Open 2 (250B)

    Solar Open 2 (250B)

    Upstage, released 2026-07-22. Hybrid MoE decoder, 250B total, 15B active, 1M context. Attention: 3:1 Kimi Delta Attention and gated GQA with NoPE.

  • Architecture diagram of Antares (1B)

    Antares (1B)

    Cisco, released 2026-07-21. Dense decoder, 1B parameters, 128K context. Attention: GQA with RoPE.

  • Architecture diagram of Laguna S 2.1 (118B)

    Laguna S 2.1 (118B)

    Poolside, released 2026-07-21. Sparse MoE decoder, 118B total, 8B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.

  • Architecture diagram of Motif 3 Beta (314B)

    Motif 3 Beta (314B)

    Motif Technologies, released 2026-07-21. Sparse MoE decoder, 314B total, 13B active, 256K context. Attention: GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention.

  • Architecture diagram of Nanbeige 4.2 (3B)

    Nanbeige 4.2 (3B)

    BOSS Zhipin, released 2026-07-21. Dense decoder, 4B parameters, 256K context. Attention: GQA with RoPE.

  • Architecture diagram of BTL-3 (27B)

    BTL-3 (27B)

    Bad Theory Labs, released 2026-07-20. Dense hybrid decoder, 27B parameters, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of Inkling (975B)

    Inkling (975B)

    Thinking Machines Lab, released 2026-07-15. Sparse MoE decoder, 975B total, 41B active, 1M context. Attention: GQA with QK-Norm and learned relative-position bias in a 5:1 sliding-window/global pattern.

  • Architecture diagram of Soofi S (30B-A3B)

    Soofi S (30B-A3B)

    SOOFI Consortium, released 2026-07-10. Hybrid MoE decoder, 32B total, 3.2B active, 1M context. Attention: Mostly Mamba-2 with 6 GQA layers and no positional embeddings (NoPE).

  • Architecture diagram of Laguna XS 2.1 (33B)

    Laguna XS 2.1 (33B)

    Poolside, released 2026-07-02. Sparse MoE decoder, 33B total, 3B active, 256K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.

  • Architecture diagram of GLM-5.2 (744B)

    GLM-5.2 (744B)

    Z.ai, released 2026-06-17. Sparse MoE decoder, 744B total, 40B active, 1M context. Attention: MLA with DeepSeek Sparse Attention and IndexShare.

  • Architecture diagram of VibeThinker-3B (3B)

    VibeThinker-3B (3B)

    Weibo, released 2026-06-15. Dense decoder, 3B parameters, 128K context. Attention: 8:1 GQA with RoPE.

  • Architecture diagram of MiniMax M3 (428B)

    MiniMax M3 (428B)

    MiniMax, released 2026-06-13. Sparse MoE decoder, 428B total, 23B active, 1M context. Attention: GQA with QK-Norm and MiniMax Sparse Attention.

  • Architecture diagram of Kimi K2.7 Code (1T)

    Kimi K2.7 Code (1T)

    Moonshot AI, released 2026-06-12. Sparse MoE decoder, 1T total, 32B active, 256K context. Attention: MLA.

  • Architecture diagram of North Mini Code (30B-A3B)

    North Mini Code (30B-A3B)

    Cohere, released 2026-06-05. Sparse MoE decoder, 30B total, 3B active, 256K context. Attention: 8:1 GQA with 3:1 sliding-window/global attention.

  • Architecture diagram of Nemotron 3 Ultra (550B-A55B)

    Nemotron 3 Ultra (550B-A55B)

    NVIDIA, released 2026-06-04. Hybrid MoE decoder, 550B total, 55B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.

  • Architecture diagram of Gemma 4 (12B)

    Gemma 4 (12B)

    Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.

  • Architecture diagram of LFM2.5 (1.2B)

    LFM2.5 (1.2B)

    Liquid AI, released 2026-05-28. Dense hybrid decoder, 1.2B parameters, 128K context. Attention: LIV convolution blocks plus GQA.

  • Architecture diagram of LFM2.5 (350M)

    LFM2.5 (350M)

    Liquid AI, released 2026-05-28. Dense hybrid decoder, 350M parameters, 128K context. Attention: LIV convolution blocks plus GQA.

  • Architecture diagram of LFM2.5 (8B-A1B)

    LFM2.5 (8B-A1B)

    Liquid AI, released 2026-05-28. Hybrid MoE decoder, 8.3B total, 1.5B active, 128K context. Attention: LIV convolution blocks plus GQA and MoE.

  • Architecture diagram of Command A+ (218B-A25B)

    Command A+ (218B-A25B)

    Cohere, released 2026-05-20. Sparse MoE decoder, 218B total, 25B active, 128K context. Attention: 16:1 GQA with 3:1 sliding-window/global attention.

  • Architecture diagram of ZAYA1-8B (8.4B)

    ZAYA1-8B (8.4B)

    Zyphra, released 2026-05-06. Sparse MoE decoder, 8.4B total, 760M active, 128K context. Attention: CCA with 4:1 GQA, RoPE, and Q/K L2 norm.

  • Architecture diagram of Granite 4.1 (30B)

    Granite 4.1 (30B)

    IBM, released 2026-04-29. Dense decoder, 30B parameters, 128K context. Attention: GQA with RoPE.

  • Architecture diagram of Laguna XS.2 (33B)

    Laguna XS.2 (33B)

    Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.

  • Architecture diagram of DeepSeek V4-Flash (284B)

    DeepSeek V4-Flash (284B)

    DeepSeek, released 2026-04-24. Sparse MoE decoder, 284B total, 13B active, 1M context. Attention: MLA-style CSA/HCA with mHC.

  • Architecture diagram of DeepSeek V4-Pro (1.6T)

    DeepSeek V4-Pro (1.6T)

    DeepSeek, released 2026-04-24. Sparse MoE decoder, 1.6T total, 49B active, 1M context. Attention: MLA-style CSA/HCA with mHC.

  • Architecture diagram of Ling 2.6 (1T)

    Ling 2.6 (1T)

    InclusionAI, released 2026-04-23. Hybrid MoE decoder, 1T total, 63B active, 256K context. Attention: Lightning Attention plus MLA.

  • Architecture diagram of Qwen3.6 (27B)

    Qwen3.6 (27B)

    Alibaba Cloud, released 2026-04-22. Dense hybrid decoder, 27B parameters, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of Xiaomi MiMo-V2.5 (310B)

    Xiaomi MiMo-V2.5 (310B)

    Xiaomi, released 2026-04-22. Sparse MoE decoder, 310B total, 15B active, 1M context. Attention: 5:1 sliding-window/global attention.

  • Architecture diagram of Xiaomi MiMo-V2.5-Pro (1.02T)

    Xiaomi MiMo-V2.5-Pro (1.02T)

    Xiaomi, released 2026-04-22. Sparse MoE decoder, 1.02T total, 42B active, 1M context. Attention: GQA with 6:1 sliding-window/global attention.

  • Architecture diagram of Kimi K2.6 (1T)

    Kimi K2.6 (1T)

    Moonshot AI, released 2026-04-20. Sparse MoE decoder, 1T total, 32B active, 256K context. Attention: MLA.

  • Architecture diagram of Qwen3.6 (35B-A3B)

    Qwen3.6 (35B-A3B)

    Alibaba Cloud, released 2026-04-15. Hybrid MoE decoder, 35B total, 3B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of GLM-5.1 (744B)

    GLM-5.1 (744B)

    Z.ai, released 2026-04-07. Sparse MoE decoder, 744B total, 40B active, 198K context. Attention: MLA with DeepSeek Sparse Attention.

  • Architecture diagram of Gemma 4 (31B)

    Gemma 4 (31B)

    Google, released 2026-04-02. Dense decoder, 31B parameters, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.

  • Architecture diagram of Gemma 4 (26B-A4B)

    Gemma 4 (26B-A4B)

    Google, released 2026-04-02. Sparse MoE decoder, 25B total, 3.8B active, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.

  • Architecture diagram of Gemma 4 (E2B)

    Gemma 4 (E2B)

    Google, released 2026-04-02. Dense decoder, 5.1B parameters, 128K context. Attention: Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention.

  • Architecture diagram of Gemma 4 (E4B)

    Gemma 4 (E4B)

    Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.

  • Architecture diagram of MiniMax M2.7 (230B)

    MiniMax M2.7 (230B)

    MiniMax, released 2026-03-18. Sparse MoE decoder, 230B total, 10B active, 192K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Mistral Small 4 (119B)

    Mistral Small 4 (119B)

    Mistral AI, released 2026-03-16. Sparse MoE decoder, 119B total, 6.6B active, 256K context. Attention: MLA.

  • Architecture diagram of Nemotron 3 Nano (4B)

    Nemotron 3 Nano (4B)

    NVIDIA, released 2026-03-16. Dense hybrid decoder, 4B parameters, 256K context. Attention: GQA with only 4 attention layers.

  • Architecture diagram of Nemotron 3 Super (120B-A12B)

    Nemotron 3 Super (120B-A12B)

    NVIDIA, released 2026-03-11. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.

  • Architecture diagram of Sarvam (30B)

    Sarvam (30B)

    Sarvam AI, released 2026-03-03. Sparse MoE decoder, 30B total, 2.4B active, 128K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Sarvam (105B)

    Sarvam (105B)

    Sarvam AI, released 2026-03-03. Sparse MoE decoder, 105B total, 10B active, 128K context. Attention: MLA with KV LayerNorm and NoPE + RoPE.

  • Architecture diagram of Qwen3.5 (397B)

    Qwen3.5 (397B)

    Alibaba Cloud, released 2026-02-16. Hybrid MoE decoder, 397B total, 17B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of Ling 2.5 (1T)

    Ling 2.5 (1T)

    InclusionAI, released 2026-02-15. Hybrid MoE decoder, 1T total, 63B active, 256K context. Attention: Lightning Attention plus MLA.

  • Architecture diagram of Tiny Aya (3.35B)

    Tiny Aya (3.35B)

    Cohere, released 2026-02-13. Dense decoder, 3.4B parameters, 8K context. Attention: GQA with 3:1 sliding-window attention.

  • Architecture diagram of MiniMax-M2.5 (230B)

    MiniMax-M2.5 (230B)

    MiniMax, released 2026-02-12. Sparse MoE decoder, 230B total, 10B active, 192K context. Attention: GQA with QK-Norm.

  • Architecture diagram of GLM-5 (744B)

    GLM-5 (744B)

    Z.ai, released 2026-02-11. Sparse MoE decoder, 744B total, 40B active, 198K context. Attention: MLA with DeepSeek Sparse Attention.

  • Architecture diagram of Nanbeige 4.1 (3B)

    Nanbeige 4.1 (3B)

    BOSS Zhipin, released 2026-02-10. Dense decoder, 3B parameters, 256K context. Attention: GQA.

  • Architecture diagram of Step 3.5 Flash (196B)

    Step 3.5 Flash (196B)

    StepFun, released 2026-02-01. Sparse MoE decoder, 196B total, 11B active, 256K context. Attention: GQA with 3:1 sliding-window attention.

  • Architecture diagram of Arcee AI Trinity Large (400B)

    Arcee AI Trinity Large (400B)

    Arcee AI, released 2026-01-27. Sparse MoE decoder, 400B total, 13B active, 512K context. Attention: GQA with gated attention and 3:1 sliding-window/global attention.

  • Architecture diagram of Kimi K2.5 (1T)

    Kimi K2.5 (1T)

    Moonshot AI, released 2026-01-27. Sparse MoE decoder, 1T total, 32B active, 256K context. Attention: MLA.

  • Architecture diagram of GLM-4.7 (355B)

    GLM-4.7 (355B)

    Z.ai, released 2025-12-22. Sparse MoE decoder, 355B total, 32B active, 198K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Xiaomi MiMo-V2-Flash (309B)

    Xiaomi MiMo-V2-Flash (309B)

    Xiaomi, released 2025-12-16. Sparse MoE decoder, 309B total, 15B active, 256K context. Attention: 5:1 sliding-window/global attention.

  • Architecture diagram of Nemotron 3 Nano (30B-A3B)

    Nemotron 3 Nano (30B-A3B)

    NVIDIA, released 2025-12-04. Hybrid MoE decoder, 30B total, 3B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.

  • Architecture diagram of Mistral Large 3 (673B)

    Mistral Large 3 (673B)

    Mistral AI, released 2025-12-02. Sparse MoE decoder, 673B total, 41B active, 256K context. Attention: MLA.

  • Architecture diagram of DeepSeek V3.2 (671B)

    DeepSeek V3.2 (671B)

    DeepSeek, released 2025-12-01. Sparse MoE decoder, 671B total, 37B active, 128K context. Attention: MLA with DeepSeek Sparse Attention.

  • Architecture diagram of INTELLECT-3 (106B)

    INTELLECT-3 (106B)

    Prime Intellect, released 2025-11-26. Sparse MoE decoder, 106B total, 12B active, 128K context. Attention: GQA.

  • Architecture diagram of OLMo 3 (32B)

    OLMo 3 (32B)

    Ai2, released 2025-11-20. Dense decoder, 32B parameters, 64K context. Attention: GQA with QK-Norm and 3:1 sliding-window/global attention.

  • Architecture diagram of OLMo 3 (7B)

    OLMo 3 (7B)

    Ai2, released 2025-11-20. Dense decoder, 7B parameters, 64K context. Attention: MHA with QK-Norm and 3:1 sliding-window/global attention.

  • Architecture diagram of Kimi Linear (48B-A3B)

    Kimi Linear (48B-A3B)

    Moonshot AI, released 2025-10-30. Hybrid MoE decoder, 48B total, 3B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA.

  • Architecture diagram of Ouro-Thinking (2.6B)

    Ouro-Thinking (2.6B)

    ByteDance, released 2025-10-29. Dense decoder, 2.6B parameters, 64K context. Attention: MHA with RoPE.

  • Architecture diagram of MiniMax M2 (230B)

    MiniMax M2 (230B)

    MiniMax, released 2025-10-23. Sparse MoE decoder, 230B total, 10B active, 192K context. Attention: GQA with QK-Norm and partial RoPE.

  • Architecture diagram of Qwen3 Next (80B-A3B)

    Qwen3 Next (80B-A3B)

    Alibaba Cloud, released 2025-09-09. Hybrid MoE decoder, 80B total, 3B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.

  • Architecture diagram of Grok 2.5 (270B)

    Grok 2.5 (270B)

    xAI, released 2025-08-22. Sparse MoE decoder, 270B parameters, 128K context. Attention: GQA.

  • Architecture diagram of Gemma 3 (270M)

    Gemma 3 (270M)

    Google, released 2025-08-14. Dense decoder, 270M parameters, 128K context. Attention: Multi-query attention with QK-Norm and 5:1 sliding-window/global attention.

  • Architecture diagram of GPT-OSS (120B)

    GPT-OSS (120B)

    OpenAI, released 2025-08-04. Sparse MoE decoder, 117B total, 5.1B active, 128K context. Attention: GQA with alternating sliding-window and global layers.

  • Architecture diagram of GPT-OSS (20B)

    GPT-OSS (20B)

    OpenAI, released 2025-08-04. Sparse MoE decoder, 21B total, 3.6B active, 128K context. Attention: GQA with alternating sliding-window and global layers.

  • Architecture diagram of Qwen3 Coder Flash (30B-A3B)

    Qwen3 Coder Flash (30B-A3B)

    Alibaba Cloud, released 2025-07-31. Sparse MoE decoder, 30B total, 3.3B active, 256K context. Attention: GQA.

  • Architecture diagram of GLM-4.5 (355B)

    GLM-4.5 (355B)

    Z.ai, released 2025-07-28. Sparse MoE decoder, 355B total, 32B active, 128K context. Attention: GQA with QK-Norm.

  • Architecture diagram of GLM-4.5-Air (106B)

    GLM-4.5-Air (106B)

    Z.ai, released 2025-07-28. Sparse MoE decoder, 106B total, 12B active, 128K context. Attention: GQA.

  • Architecture diagram of Kimi K2 (1T)

    Kimi K2 (1T)

    Moonshot AI, released 2025-07-10. Sparse MoE decoder, 1T total, 32B active, 128K context. Attention: MLA.

  • Architecture diagram of SmolLM3 (3B)

    SmolLM3 (3B)

    Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.

  • Architecture diagram of Qwen3 (0.6B)

    Qwen3 (0.6B)

    Alibaba Cloud, released 2025-04-28. Dense decoder, 600M parameters, 32K context. Attention: GQA.

  • Architecture diagram of Qwen3 (235B-A22B)

    Qwen3 (235B-A22B)

    Alibaba Cloud, released 2025-04-28. Sparse MoE decoder, 235B total, 22B active, 128K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Qwen3 (30B-A3B)

    Qwen3 (30B-A3B)

    Alibaba Cloud, released 2025-04-28. Sparse MoE decoder, 30B total, 3B active, 128K context. Attention: GQA.

  • Architecture diagram of Qwen3 (32B)

    Qwen3 (32B)

    Alibaba Cloud, released 2025-04-28. Dense decoder, 32B parameters, 128K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Qwen3 (4B)

    Qwen3 (4B)

    Alibaba Cloud, released 2025-04-28. Dense decoder, 4B parameters, 32K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Qwen3 (8B)

    Qwen3 (8B)

    Alibaba Cloud, released 2025-04-28. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm.

  • Architecture diagram of Llama 4 Maverick (400B)

    Llama 4 Maverick (400B)

    Meta, released 2025-04-05. Sparse MoE decoder, 400B total, 17B active, 1M context. Attention: GQA.

  • Architecture diagram of Mistral Small 3.1 (24B)

    Mistral Small 3.1 (24B)

    Mistral AI, released 2025-03-18. Dense decoder, 24B parameters, 128K context. Attention: Standard GQA.

  • Architecture diagram of xLSTM (7B)

    xLSTM (7B)

    NXAI, released 2025-03-17. Recurrent decoder, 7B parameters, — context. Attention: No self-attention; mLSTM recurrent layers with matrix memory.

  • Architecture diagram of Gemma 3 (27B)

    Gemma 3 (27B)

    Google, released 2025-03-11. Dense decoder, 27B parameters, 128K context. Attention: GQA with QK-Norm and 5:1 sliding-window/global attention.

  • Architecture diagram of DeepSeek R1 (671B)

    DeepSeek R1 (671B)

    DeepSeek, released 2025-01-20. Sparse MoE decoder, 671B total, 37B active, 128K context. Attention: MLA.

  • Architecture diagram of DeepSeek V3 (671B)

    DeepSeek V3 (671B)

    DeepSeek, released 2024-12-26. Sparse MoE decoder, 671B total, 37B active, 128K context. Attention: MLA.

  • Architecture diagram of Phi-4 (14B)

    Phi-4 (14B)

    Microsoft, released 2024-12-12. Dense decoder, 14B parameters, 16K context. Attention: GQA with RoPE.

  • Architecture diagram of OLMo 2 (7B)

    OLMo 2 (7B)

    Ai2, released 2024-11-25. Dense decoder, 7B parameters, 4K context. Attention: MHA with QK-Norm.

  • Architecture diagram of Llama 3.2 (1B)

    Llama 3.2 (1B)

    Meta, released 2024-09-25. Dense decoder, 1B parameters, 128K context. Attention: GQA.

  • Architecture diagram of Llama 3.2 (3B)

    Llama 3.2 (3B)

    Meta, released 2024-09-25. Dense decoder, 3B parameters, 128K context. Attention: GQA.

  • Architecture diagram of Llama 3 (8B)

    Llama 3 (8B)

    Meta, released 2024-04-18. Dense decoder, 8B parameters, 8K context. Attention: GQA with RoPE.

  • Architecture diagram of GPT-2 XL (1.5B)

    GPT-2 XL (1.5B)

    OpenAI, released 2019-11-05. Dense decoder, 1.5B parameters, 1K context. Attention: MHA with learned absolute positional embeddings.

Search the atlas

Jump to a model, concept or page