Architecture diagram of Llama 3 (8B) — Figure 10: Llama 3 8B · diagram by Sebastian Raschka · click to zoom

DenseMeta 🇺🇸 · April 18, 2024

Llama 3 (8B)

Reference dense Llama stack used to contrast OLMo 2's normalization and attention choices.

Key detail

Pre-norm baseline; wider than OLMo 2 at a similar scale.

Parameters

8B

Context

8K

KV / token

128 KiB

Fact sheet

Scale: 8B parameters
Context window: 8,192 tokens
Decoder type: Dense
Attention: GQA with RoPE
Layer mix: 32 GQA
KV cache / token (bf16): 128 KiB · Moderate
License: Llama 3 Community License Agreement
Released: April 18, 2024

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

128 KiB

Smaller than 37% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

4.8

Scientific: 2.5

Architecture concepts

GQAAttention

Sources & resources

View in article From scratch config.json License Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

Qwen3 (8B)
Alibaba Cloud, released 2025-04-28. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm.
Gemma 4 (E4B)
Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Gemma 4 (12B)
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Phi-4 (14B)
Microsoft, released 2024-12-12. Dense decoder, 14B parameters, 16K context. Attention: GQA with RoPE.