Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.
Sparse MoEMeta 🇺🇸 · October 15, 2026
Muse Spark Open (816B-A42B)
Meta's multimodal MoE model combines 62 latent-expert blocks with gated GQA, local-only RoPE, global NoPE, and learned residual gates; the figure focuses on the text decoder.
Key detail
Selects 8 of 256 experts, compresses the expert path from 8,192 to 4,096 dimensions, normalizes each expert output before combining, and uses no shared expert.Total params
816B
Active / token
42B
Context
1M
Fact sheet
- Scale
- 816B total, 42B active (5.1% active)
- Context window
- 1,048,576 tokens
- Decoder type
- Sparse MoE
- Attention
- Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention
- Layer mix
- 46 sliding-window + 16 global gated GQA
- KV cache / token (bf16)
- 248 KiB · High
- License
- Muse Model License
- Released
Efficiency
Parameters active per token
5.1%
42B of 816B weights run for each token.
AA Intelligence Index v5.0
Source24.7
- Knowledge Work
- 37
- Coding
- 7.8
- Science
- 12.1
- General
- 41.7
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Arcee AI, released 2026-01-27. Sparse MoE decoder, 400B total, 13B active, 512K context. Attention: GQA with gated attention and 3:1 sliding-window/global attention.
StepFun, released 2026-09-20. Sparse MoE decoder, 600B total, 27B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention.
Aleph Alpha, released 2026-10-03. Sparse MoE decoder, 78B total, 3.5B active, 256K context. Attention: GQA with QK-Norm and 4:1 sliding-window/global attention.