# Attention Residuals (AttnRes)

> Learned, attention-style weights over earlier layer outputs replace the plain residual sum.

Category: Residual stream

A standard residual connection adds every earlier update with weight one. AttnRes lets each layer attend over earlier layer outputs and weight their contributions.

Full and block variants exist; Kimi Linear already used the idea. Reported gains are consistent but modest, for a few percent extra training and inference cost.

**Trade-off:** Smarter depth-wise information flow for a small overhead.

## Models using AttnRes (3)

- [Ember-1 (2.8T)](https://llm-atlas-7or.pages.dev/models/ember-1.md) — Fireworks AI, 2026-09-23
- [Kimi K3 (2.8T)](https://llm-atlas-7or.pages.dev/models/kimi-k3.md) — Moonshot AI, 2026-07-27
- [Kimi Linear (48B-A3B)](https://llm-atlas-7or.pages.dev/models/kimi-linear-48b-a3b.md) — Moonshot AI, 2025-10-30

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/attention-residuals/
Web page: https://llm-atlas-7or.pages.dev/concepts/attention-residuals/