JetBrains, released 2026-06-01. Sparse MoE decoder, 12B total, 2.5B active, 128K context. Attention: GQA with 3:1 sliding-window/full attention.
Sparse MoEZyphra 🇺🇸 · May 6, 2026
ZAYA1-8B (8.4B)
Zyphra's compact reasoning MoE pairs CCA/GQA attention updates with top-1 routed expert FFN updates to keep fewer than 1B parameters active per token.
Key detail
Key detail is CCA/GQA attention, which performs attention in a compressed latent space while keeping a 4:1 GQA layout. Sidenote: the config lists 80 alternating layer entries, equivalent to 40 CCA/GQA attention updates paired with 40 top-1 MoE FFN updates.Total params
8.4B
Active / token
760M
Context
128K
Fact sheet
- Scale
- 8.4B total, 760M active/token (9% active)
- Context window
- 131,072 tokens
- Decoder type
- Sparse MoE
- Attention
- CCA with 4:1 GQA, RoPE, and Q/K L2 norm
- Layer mix
- 40 CCA/GQA attention + 40 top-1 MoE FFN entries
- KV cache / token (bf16)
- 40 KiB · Low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
9.0%
760M of 8.4B weights run for each token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
OpenAI, released 2025-08-04. Sparse MoE decoder, 21B total, 3.6B active, 128K context. Attention: GQA with alternating sliding-window and global layers.
Google, released 2026-04-02. Sparse MoE decoder, 25B total, 3.8B active, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Alibaba Cloud, released 2025-04-28. Sparse MoE decoder, 30B total, 3B active, 128K context. Attention: GQA.