Architecture diagram of Qwen3 (32B) — Figure 45: Qwen3 32B · diagram by Sebastian Raschka · click to zoom

DenseAlibaba Cloud 🇨🇳 · April 28, 2025

Qwen3 (32B)

Large dense Qwen3 model that serves as the clearest like-for-like comparison for OLMo 3 32B.

Key detail

Reference dense Qwen stack with QK-Norm and 8 KV heads.

Parameters

32B

Context

128K

KV / token

256 KiB

Fact sheet

Scale: 32B parameters
Context window: 128,000 tokens
Decoder type: Dense
Attention: GQA with QK-Norm
Layer mix: 64 GQA
KV cache / token (bf16): 256 KiB · High
License: Apache License 2.0
Released: April 28, 2025

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

256 KiB

Smaller than 12% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

7.3

Architecture concepts

Sources & resources

View in article From scratch config.json License Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

OLMo 3 (32B)
Ai2, released 2025-11-20. Dense decoder, 32B parameters, 64K context. Attention: GQA with QK-Norm and 3:1 sliding-window/global attention.
Gemma 4 (31B)
Google, released 2026-04-02. Dense decoder, 31B parameters, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Muse Glimmer (30B)
Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
Gemma 3 (27B)
Google, released 2025-03-11. Dense decoder, 27B parameters, 128K context. Attention: GQA with QK-Norm and 5:1 sliding-window/global attention.