Skip to content

Sparse & compressed attention · QSA

Qwen Sparse Attention

Like DSA, but the indexer picks whole four-token blocks rather than single tokens.

The indexer averages keys within each complete four-token block, scores the pooled keys against the query, and expands the winning blocks back into their tokens. The current incomplete block is always visible.

Selecting contiguous blocks is friendlier to memory access patterns than scattering over individual token positions.

Read the full explainer Browse all 1 models
Qwen3.8-Flash-Next architecture showing Qwen Sparse Attention. Figure from Sebastian Raschka’s explainer.

Adoption over time

Models in the atlas using QSA, per half-year of release.

Loading

Used by

1

models in the atlas

Earliest: Qwen3.8-Flash-Next (125B-A6B) (Aug 26, 2026)

Often paired with

Models using QSA

  • Architecture diagram of Qwen3.8-Flash-Next (125B-A6B)

    Qwen3.8-Flash-Next (125B-A6B)

    Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).

Search the atlas

Jump to a model, concept or page