Architecture diagram of Qwen3 (4B) — Figure 21: Qwen3 4B · diagram by Sebastian Raschka · click to zoom

DenseAlibaba Cloud 🇨🇳 · April 28, 2025

Qwen3 (4B)

Mid-size dense Qwen3 model used here as a clean baseline against SmolLM3 and Tiny Aya.

Key detail

Compact Qwen3 dense stack with QK-Norm and a 151k vocabulary.

Parameters

4B

Context

32K

KV / token

144 KiB

Fact sheet

Scale: 4B parameters
Context window: 32,768 tokens
Decoder type: Dense
Attention: GQA with QK-Norm
Layer mix: 36 GQA
KV cache / token (bf16): 144 KiB · Moderate
License: Apache License 2.0
Released: April 28, 2025

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

144 KiB

Smaller than 34% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

6.6

Architecture concepts

Sources & resources

View in article From scratch config.json License Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

Dory1.0 (4B)
NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.
Qwen3 (8B)
Alibaba Cloud, released 2025-04-28. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm.
Gemma 4 (E4B)
Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Gemma 4 (12B)
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.