Sarvam AI, released 2026-03-03. Sparse MoE decoder, 105B total, 10B active, 128K context. Attention: MLA with KV LayerNorm and NoPE + RoPE.
Sparse MoEMistral AI 🇫🇷 · March 16, 2026
Mistral Small 4 (119B)
Multimodal Mistral Small refresh that jumps from the older dense 24B stack to an MLA-based sparse MoE design.
Key detail
Uses 128 experts with 4 routed plus 1 shared expert active per token while unifying instruct, reasoning, and vision.Total params
119B
Active / token
6.6B
Context
256K
Fact sheet
- Scale
- 119B total, 6.63B active (5.6% active)
- Context window
- 256,000 tokens
- Decoder type
- Sparse MoE
- Attention
- MLA
- Layer mix
- 36 MLA
- KV cache / token (bf16)
- 22.5 KiB · Very low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
5.6%
6.6B of 119B weights run for each token.
AA Intelligence Index v5.0
Source4.3
- Knowledge Work
- 0.2
- Coding
- 0
- Science
- 0
- General
- 17
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Meituan, released 2026-01-28. Sparse MoE decoder, 69B total, 3B active, 256K context. Attention: MLA with RoPE + NoPE.
DeepSeek, released 2026-04-24. Sparse MoE decoder, 284B total, 13B active, 1M context. Attention: MLA-style CSA/HCA with mHC.
Motif Technologies, released 2026-07-21. Sparse MoE decoder, 314B total, 13B active, 256K context. Attention: GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention.