Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.
Sparse MoEStepFun 🇨🇳 · September 20, 2026
Step-5 Preview (600B)
StepFun's 92-layer MoE pairs 512-token sliding-window attention with sparse GQA for a 1M-token context window; the figure shows the text decoder.
Key detail
Uses top-512 sparse indexing with block size 8, 352 routed experts with 8 active plus one shared expert, and MTP-3. The first three and final backbone layers use dense FFNs with intermediate size 13,824.Total params
600B
Active / token
27B
Context
1M
Fact sheet
- Scale
- 600B total, 27B active (4.5% active)
- Context window
- 1,048,576 tokens
- Decoder type
- Sparse MoE
- Attention
- Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention
- Layer mix
- 69 sliding-window + 23 sparse GQA + 3 MTP layers
- KV cache / token (bf16)
- N/A
- License
- N/A
- Released
Efficiency
Parameters active per token
4.5%
27B of 600B weights run for each token.
KV cache per token (bf16)
N/A
Not reported for this model.
AA Intelligence Index v5.0
Source28.5
- Knowledge Work
- 40.9
- Coding
- 20.5
- Science
- 14.4
- General
- 38
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Meta, released 2026-10-15. Sparse MoE decoder, 816B total, 42B active, 1M context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
Arcee AI, released 2026-01-27. Sparse MoE decoder, 400B total, 13B active, 512K context. Attention: GQA with gated attention and 3:1 sliding-window/global attention.
Thinking Machines Lab, released 2026-07-15. Sparse MoE decoder, 975B total, 41B active, 1M context. Attention: GQA with QK-Norm and learned relative-position bias in a 5:1 sliding-window/global pattern.