Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Sparse MoEPoolside 🇺🇸 · July 2, 2026
Laguna XS 2.1 (33B)
Poolside's 33B Laguna refresh extends the sparse MoE architecture to 262k context while keeping about 3B parameters active per token.
Key detail
Uses 256 routed experts with top-8 routing plus one shared expert, a dense first layer, and per-layer query-head counts.Total params
33B
Active / token
3B
Context
256K
Fact sheet
- Scale
- 33B total, 3B active (9.1% active)
- Context window
- 262,144 tokens
- Decoder type
- Sparse MoE
- Attention
- Gated GQA with QK-Norm and 3:1 sliding-window/global attention
- Layer mix
- 30 sliding-window + 10 global GQA
- KV cache / token (bf16)
- 160 KiB · Moderate
- License
- OpenMDW License 1.1
- Released
Efficiency
Parameters active per token
9.1%
3B of 33B weights run for each token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Poolside, released 2026-07-21. Sparse MoE decoder, 118B total, 8B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Recursive, released 2026-10-18. Sparse MoE decoder, 40B total, 2.5B active, 256K context. Attention: GQA with QK-Norm, learned attention sinks, and sliding-window/global attention.
Google, released 2026-04-02. Sparse MoE decoder, 25B total, 3.8B active, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.