Architecture diagram of Llama 3.2 (3B) — Figure 29: Llama 3.2 3B · diagram by Sebastian Raschka · click to zoom

DenseMeta 🇺🇸 · September 25, 2024

Llama 3.2 (3B)

Small Llama baseline used to show how closely Nanbeige follows the mainstream compact decoder recipe.

Key detail

Reference small-model Llama architecture with tied embeddings.

Parameters

3B

Context

128K

KV / token

112 KiB

Fact sheet

Scale: 3B parameters
Context window: 128,000 tokens
Decoder type: Dense
Attention: GQA
Layer mix: 28 GQA
KV cache / token (bf16): 112 KiB · Moderate
License: Llama 3.2 Community License Agreement
Released: September 25, 2024

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

112 KiB

Smaller than 41% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

5.7

Architecture concepts

GQAAttention

Sources & resources

View in article License Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

SmolLM3 (3B)
Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.
Nanbeige 4.1 (3B)
BOSS Zhipin, released 2026-02-10. Dense decoder, 3B parameters, 256K context. Attention: GQA.
VibeThinker-3B (3B)
Weibo, released 2026-06-15. Dense decoder, 3B parameters, 128K context. Attention: 8:1 GQA with RoPE.
Tiny Aya (3.35B)
Cohere, released 2026-02-13. Dense decoder, 3.4B parameters, 8K context. Attention: GQA with 3:1 sliding-window attention.