Architecture diagram of SmolLM3 (3B) — Figure 21: SmolLM3 3B · diagram by Sebastian Raschka · click to zoom

DenseHugging Face 🇺🇸 · June 19, 2025

SmolLM3 (3B)

Compact dense model that experiments with leaving out positional encodings in selected layers.

Key detail

Every fourth layer omits RoPE to test a NoPE-style cadence.

Parameters

3B

Context

128K

KV / token

72 KiB

Fact sheet

Scale: 3B parameters
Context window: 131,072 tokens
Decoder type: Dense
Attention: GQA with periodic NoPE layers
Layer mix: 36 GQA
KV cache / token (bf16): 72 KiB · Low
License: Apache License 2.0
Released: June 19, 2025

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

72 KiB

Smaller than 53% of models in the atlas. How it’s calculated

Architecture concepts

Sources & resources

View in article config.json Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

Tiny Aya (3.35B)
Cohere, released 2026-02-13. Dense decoder, 3.4B parameters, 8K context. Attention: GQA with 3:1 sliding-window attention.
Muse Glimmer (30B)
Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
Llama 3.2 (3B)
Meta, released 2024-09-25. Dense decoder, 3B parameters, 128K context. Attention: GQA.
Nanbeige 4.1 (3B)
BOSS Zhipin, released 2026-02-10. Dense decoder, 3B parameters, 256K context. Attention: GQA.