Sparse MoE (causal encoder-decoder)DeepSeek 🇨🇳 · September 10, 2026

DeepSeek V4.1-Flash (552B)

Causal encoder-decoder MoE with CSA2 cache sharing, Engram memory, and a million-token context. The figure shows the text backbone.

Key detail

Decoder global KV comes from final encoder states. Single-Pass mHC mixes four residual streams around each attention and MoE sublayer.

Total params

552B

Active / token

Context

Fact sheet

Scale: 552B backbone + 196B Engram; 8B active prefill, 16B active decode (2.9% of backbone)
Context window: 1,048,576 tokens
Decoder type: Sparse MoE (causal encoder-decoder)
Attention: MLA-style CSA2 + SWA with cross-layer KV and sparse-index reuse
Layer mix: 20 encoder + 20 decoder layers; 2 SWA-only + 38 CSA2/SWA
KV cache / token (bf16): 3.125 KiB · Very low
License: MIT License
Released: September 10, 2026

Parameters active per token

1.4%

8B of 552B weights run for each token.

KV cache per token (bf16)

3.125 KiB

Smaller than 99% of models in the atlas. How it’s calculated

39.5

Use the ← and → keys to step through releases.

Motif 3 Beta (314B)
Motif Technologies, released 2026-07-21. Sparse MoE decoder, 314B total, 13B active, 256K context. Attention: GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention.
Step-5 Preview (600B)
StepFun, released 2026-09-20. Sparse MoE decoder, 600B total, 27B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention.
Reflection Beam (501B)
Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.
Arcee AI Trinity Large (400B)
Arcee AI, released 2026-01-27. Sparse MoE decoder, 400B total, 13B active, 512K context. Attention: GQA with gated attention and 3:1 sliding-window/global attention.