Sparse MoEGoogle 🇺🇸 · April 2, 2026

Gemma 4 (26B-A4B)

Sparse Gemma 4 variant that keeps the local:global attention backbone while swapping dense FFNs for MoE layers.

Key detail

Uses 128 total experts with only 8 routed plus 1 shared expert active per token.

Total params

25B

Active / token

3.8B

Context

256K

Fact sheet

Scale: 25.2B total, 3.8B active (15.1% active)
Context window: 256,000 tokens
Decoder type: Sparse MoE
Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
Layer mix: 25 sliding-window + 5 global
KV cache / token (bf16): 210 KiB · High
License: Apache License 2.0
Released: April 2, 2026

Parameters active per token

15%

3.8B of 25B weights run for each token.

KV cache per token (bf16)

210 KiB

Smaller than 22% of models in the atlas. How it’s calculated

13.1

Use the ← and → keys to step through releases.

Laguna XS.2 (33B)
Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Laguna XS 2.1 (33B)
Poolside, released 2026-07-02. Sparse MoE decoder, 33B total, 3B active, 256K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Recursive Flint (40B-A2.5B)
Recursive, released 2026-10-18. Sparse MoE decoder, 40B total, 2.5B active, 256K context. Attention: GQA with QK-Norm, learned attention sinks, and sliding-window/global attention.
Kolibri-1 (78B)
Aleph Alpha, released 2026-10-03. Sparse MoE decoder, 78B total, 3.5B active, 256K context. Attention: GQA with QK-Norm and 4:1 sliding-window/global attention.