Skip to content

Residual stream · GR

Gated Residuals

Gated read/write modules over four residual streams, without mHC's mixing matrix.

Before a sublayer, a GR Read module normalises each of four residual streams, gates them element-wise and averages them into a normal-width input. Afterwards, GR Write adds the output back to all four streams with learned gates.

It was introduced with Qwen3.8-Flash-Next and is related in spirit to mHC.

Read the full explainer Browse all 1 models
Qwen3.8-Flash-Next architecture showing GR Read and GR Write. Figure from Sebastian Raschka’s explainer.

Adoption over time

Models in the atlas using GR, per half-year of release.

Loading

Used by

1

models in the atlas

Earliest: Qwen3.8-Flash-Next (125B-A6B) (Aug 26, 2026)

Often paired with

Models using GR

  • Architecture diagram of Qwen3.8-Flash-Next (125B-A6B)

    Qwen3.8-Flash-Next (125B-A6B)

    Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).

Search the atlas

Jump to a model, concept or page