# Dory1.0 (4B)

> NVIDIA's dense looped transformer applies a shared 20-block core twice between 8-block input and output stacks, using normalized GPT (nGPT) residual updates.

| Property | Value |
| --- | --- |
| Lab | NVIDIA (United States) |
| Released | 2026-10-16 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 4B parameters |
| Context window | 262,144 tokens |
| Attention | Gated GQA; 3:1 sliding-window/full, RoPE |
| Layer mix | 8 input + 20 shared × 2 passes + 8 output (56 applications) |
| KV cache / token (bf16) | 224 KiB (High) |
| License | OpenMDW 1.1 |

**Key detail:** Reinjects the input-stack output before each recurrent pass. The 36 distinct blocks give 56 block applications by default, with L2-normalized residual updates and a 1,024-token attention window.

## Architecture concepts

- [Looped Transformer (Looped depth)](https://llm-atlas-7or.pages.dev/concepts/looped-depth-sharing.md): The same blocks are applied several times, reusing weights across depth.
- [Grouped-Query Attention (GQA)](https://llm-atlas-7or.pages.dev/concepts/gqa.md): Several query heads share one key/value head, shrinking the KV cache with little quality loss.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [QK-Norm (QK-Norm)](https://llm-atlas-7or.pages.dev/concepts/qk-norm.md): RMSNorm applied to queries and keys before attention to keep logits well-behaved.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/dory-1-0-4b.webp
- config.json: https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/main/config.json
- License: https://openmdw.ai/license/1-1/
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-dory1-0-4b
- Web page: https://llm-atlas-7or.pages.dev/models/dory1-0-4b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/dory1-0-4b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._