Architecture diagram of Qwen3 (30B-A3B) — Figure 27: Qwen3 30B-A3B · diagram by Sebastian Raschka · click to zoom

Sparse MoEAlibaba Cloud 🇨🇳 · April 28, 2025

Qwen3 (30B-A3B)

Small sparse Qwen3 model that sits close to GPT-OSS in active size but uses a deeper stack.

Key detail

Deeper, narrower MoE alternative to GPT-OSS without a shared expert.

Total params

30B

Active / token

3B

Context

128K

Fact sheet

Scale: 30B total, 3B active (10% active)
Context window: 128,000 tokens
Decoder type: Sparse MoE
Attention: GQA
Layer mix: 48 GQA
KV cache / token (bf16): 96 KiB · Moderate
License: Apache License 2.0
Released: April 28, 2025

Parameters active per token

10%

3B of 30B weights run for each token.

KV cache per token (bf16)

96 KiB

Smaller than 45% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

6.6

Scientific: 2.3

Architecture concepts

Sources & resources

View in article config.json License Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

Qwen3 Coder Flash (30B-A3B)
Alibaba Cloud, released 2025-07-31. Sparse MoE decoder, 30B total, 3.3B active, 256K context. Attention: GQA.
Sarvam (30B)
Sarvam AI, released 2026-03-03. Sparse MoE decoder, 30B total, 2.4B active, 128K context. Attention: GQA with QK-Norm.
North Mini Code (30B-A3B)
Cohere, released 2026-06-05. Sparse MoE decoder, 30B total, 3B active, 256K context. Attention: 8:1 GQA with 3:1 sliding-window/global attention.
Laguna XS.2 (33B)
Poolside, released 2026-04-28. Sparse MoE decoder, 33B total, 3B active, 128K context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/global attention.