Aleph Alpha, released 2026-10-03. Sparse MoE decoder, 78B total, 3.5B active, 256K context. Attention: GQA with QK-Norm and 4:1 sliding-window/global attention.
Sparse MoESarvam AI 🇮🇳 · March 3, 2026
Sarvam (105B)
Larger Sarvam variant keeps the sparse MoE layout but switches from GQA to MLA.
Key detail
Large vocabulary and strong Indic language support carried into the larger MLA-based sparse MoE variant.Total params
105B
Active / token
10B
Context
128K
Fact sheet
- Scale
- 105B total, 10.3B active (9.8% active)
- Context window
- 131,072 tokens
- Decoder type
- Sparse MoE
- Attention
- MLA with KV LayerNorm and NoPE + RoPE
- Layer mix
- 32 MLA
- KV cache / token (bf16)
- 36 KiB · Low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
9.8%
10B of 105B weights run for each token.
AA Intelligence Index v4.3.2
Source8.8
- Scientific
- 5.5
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Cohere, released 2026-05-20. Sparse MoE decoder, 218B total, 25B active, 128K context. Attention: 16:1 GQA with 3:1 sliding-window/global attention.
Recursive, released 2026-10-18. Sparse MoE decoder, 40B total, 2.5B active, 256K context. Attention: GQA with QK-Norm, learned attention sinks, and sliding-window/global attention.
Z.ai, released 2025-12-22. Sparse MoE decoder, 355B total, 32B active, 198K context. Attention: GQA with QK-Norm.