Skip to content

Residual stream · Looped depth

Looped Transformer

The same blocks are applied several times, reusing weights across depth.

With L distinct blocks applied T times, the model gets L × T effective layers for the parameter cost of L. It is depth-wise weight tying, sometimes called recurrent depth.

Nanbeige 4.2 uses two fixed passes, Ouro uses four passes with a learned exit gate, and Dory1.0 loops a shared middle stack.

Read the full explainer Browse all 3 models
Nanbeige 4.2 architecture with a loop around the shared transformer stack. Figure from Sebastian Raschka’s explainer.

Adoption over time

Models in the atlas using Looped depth, per half-year of release.

Loading

Used by

3

models in the atlas

Earliest: Ouro-Thinking (2.6B) (Oct 29, 2025)

Often paired with

Models using Looped depth

  • Architecture diagram of Dory1.0 (4B)

    Dory1.0 (4B)

    NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.

  • Architecture diagram of Nanbeige 4.2 (3B)

    Nanbeige 4.2 (3B)

    BOSS Zhipin, released 2026-07-21. Dense decoder, 4B parameters, 256K context. Attention: GQA with RoPE.

  • Architecture diagram of Ouro-Thinking (2.6B)

    Ouro-Thinking (2.6B)

    ByteDance, released 2025-10-29. Dense decoder, 2.6B parameters, 64K context. Attention: MHA with RoPE.

Search the atlas

Jump to a model, concept or page