DeepSeek, released 2025-01-20. Sparse MoE decoder, 671B total, 37B active, 128K context. Attention: MLA.
Sparse MoEDeepSeek 🇨🇳 · December 26, 2024
DeepSeek V3 (671B)
DeepSeek's flagship template kicked off the recent wave of large open MoE models.
Key detail
Uses a dense prefix, shared expert, and MTP-1 objective to keep a very large model practical.Total params
671B
Active / token
37B
Context
128K
Fact sheet
- Scale
- 671B total, 37B active (5.5% active)
- Context window
- 128,000 tokens
- Decoder type
- Sparse MoE
- Attention
- MLA
- Layer mix
- 61 MLA
- KV cache / token (bf16)
- 68.6 KiB · Low
- License
- DeepSeek License Agreement v1.0
- Released
Efficiency
Parameters active per token
5.5%
37B of 671B weights run for each token.
AA Intelligence Index v4.3.2
Source8.5
- General
- 15.3
- Scientific
- 1.5
- Coding
- 17.9
- Agents
- 0.1
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
DeepSeek, released 2025-12-01. Sparse MoE decoder, 671B total, 37B active, 128K context. Attention: MLA with DeepSeek Sparse Attention.
Z.ai, released 2026-02-11. Sparse MoE decoder, 744B total, 40B active, 198K context. Attention: MLA with DeepSeek Sparse Attention.
Z.ai, released 2026-04-07. Sparse MoE decoder, 744B total, 40B active, 198K context. Attention: MLA with DeepSeek Sparse Attention.