Arcee AI, released 2026-01-27. Sparse MoE decoder, 400B total, 13B active, 512K context. Attention: GQA with gated attention and 3:1 sliding-window/global attention.
Sparse MoECohere 🇨🇦 · May 20, 2026
Command A+ (218B-A25B)
Cohere's open sparse MoE model scales the Command A parallel transformer block to 218B total parameters while keeping 25B active per token.
Key detail
Runs attention and MoE in parallel, which Cohere reports gives similar quality to a sequential block with better throughput.Total params
218B
Active / token
25B
Context
128K
Fact sheet
- Scale
- 218B total, 25B active/token (11.5% active)
- Context window
- 128K input, 64K output tokens
- Decoder type
- Sparse MoE
- Attention
- 16:1 GQA with 3:1 sliding-window/global attention
- Layer mix
- 24 sliding-window + 8 global GQA/MoE
- KV cache / token (bf16)
- 128 KiB · Moderate
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
11%
25B of 218B weights run for each token.
AA Intelligence Index v5.0
Source6.2
- Knowledge Work
- 0.6
- Coding
- 0.2
- Science
- 1
- General
- 23.2
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Sarvam AI, released 2026-03-03. Sparse MoE decoder, 105B total, 10B active, 128K context. Attention: MLA with KV LayerNorm and NoPE + RoPE.
Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.
Aleph Alpha, released 2026-10-03. Sparse MoE decoder, 78B total, 3.5B active, 256K context. Attention: GQA with QK-Norm and 4:1 sliding-window/global attention.