Moonshot AI, released 2026-07-27. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.
Sparse hybridFireworks AI 🇺🇸 · September 23, 2026
Ember-1 (2.8T)
Fireworks' Kimi K3 derivative, post-trained for shorter reasoning traces. Architecture details follow the published Kimi K3 base model; the diagram focuses on its text backbone.
Key detail
Inherits Kimi K3's eight Block Attention Residual groups and Stable LatentMoE, with one dense layer followed by 92 MoE layers.Total params
2.8T
Active / token
104B
Context
1M
Fact sheet
- Scale
- 2.8T total, 104B active (3.7% active)
- Context window
- 1,048,576 tokens
- Decoder type
- Sparse hybrid
- Attention
- 3:1 Kimi Delta Attention and Gated MLA with NoPE
- Layer mix
- 69 KDA + 24 Gated MLA
- KV cache / token (bf16)
- 27 KiB · Low
- License
- N/A
- Released
Efficiency
Parameters active per token
3.7%
104B of 2.8T weights run for each token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
InclusionAI, released 2026-10-13. Hybrid MoE decoder, 560B total, 25B active, 1M context. Attention: 7:1 Kimi Delta Attention and gated MLA.
Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.
Upstage, released 2026-07-22. Hybrid MoE decoder, 250B total, 15B active, 1M context. Attention: 3:1 Kimi Delta Attention and gated GQA with NoPE.