Moonshot AI, released 2026-07-27. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.
Sparse hybridMoonshot AI 🇨🇳 · October 30, 2025
Kimi Linear (48B-A3B)
Linear-attention hybrid that keeps a transformer backbone but replaces most full-attention layers.
Key detail
Uses NoPE in MLA layers, channel-wise KDA gating, and kernel-4 ShortConv preprocessing for Q, K, and V in KDA layers.Total params
48B
Active / token
3B
Context
1M
Fact sheet
- Scale
- 48B total, 3B active (6.3% active)
- Context window
- 1,000,000 tokens
- Decoder type
- Sparse hybrid
- Attention
- 3:1 Kimi Delta Attention and MLA
- Layer mix
- 7 MLA + 20 Kimi Delta Attention
- KV cache / token (bf16)
- 7.9 KiB · Very low
- License
- MIT License
- Released
Efficiency
Parameters active per token
6.3%
3B of 48B weights run for each token.
AA Intelligence Index v4.3.2
Source7.3
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Fireworks AI, released 2026-09-23. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.
Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.
InclusionAI, released 2026-08-04. Hybrid MoE decoder, 124B total, 5.1B active, 256K context. Attention: 5:1 Kimi Delta Attention and gated MLA.