Recursive, released 2026-10-18. Sparse MoE decoder, 40B total, 2.5B active, 256K context. Attention: GQA with QK-Norm, learned attention sinks, and sliding-window/global attention.
Sparse MoEAleph Alpha 🇩🇪 · October 3, 2026
Kolibri-1 (78B)
Aleph Alpha's German-English reasoning model uses 50 MoE blocks with a 4:1 mix of sliding-window and global GQA.
Key detail
Uses top-6 routing over 384 experts plus one shared expert, with sandwich RMSNorm, RoPE in local layers, and NoPE in global layers.Total params
78B
Active / token
3.5B
Context
256K
Fact sheet
- Scale
- 78B total, 3.46B active (4.4% active)
- Context window
- 262,144 native; 1,048,576 extrapolated tokens
- Decoder type
- Sparse MoE
- Attention
- GQA with QK-Norm and 4:1 sliding-window/global attention
- Layer mix
- 40 sliding-window + 10 global GQA
- KV cache / token (bf16)
- 100 KiB · Moderate
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
4.4%
3.5B of 78B weights run for each token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Arcee AI, released 2026-01-27. Sparse MoE decoder, 400B total, 13B active, 512K context. Attention: GQA with gated attention and 3:1 sliding-window/global attention.
Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.
Meta, released 2026-10-15. Sparse MoE decoder, 816B total, 42B active, 1M context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.