Poolside, released 2026-07-21. Sparse MoE decoder, 118B total, 8B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Attention · Attention budgeting
Layer-wise attention budgeting
Different layers get different attention budgets: head counts, windows or full context.
Rather than giving every layer an identical attention setup, the model spends more attention capacity in some layers and less in others.
Laguna XS.2 is the reference example: most of its 40 layers use a 512-token sliding window, while a minority keep full attention, and the head counts differ between the two kinds of layer.
Adoption over time
Models in the atlas using Attention budgeting, per half-year of release.
Often paired with
Models using Attention budgeting
Poolside, released 2026-07-02. Sparse MoE decoder, 33B total, 3B active, 256K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.
Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.