Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
DenseGoogle 🇺🇸 · June 3, 2026
Gemma 4 (12B)
Dense Gemma 4 12B Unified uses an encoder-free multimodal input path, while its text decoder keeps the 31B-style local-global attention recipe.
Key detail
Figure shows the 48-layer text decoder; the full 12B Unified model also projects audio, image, and video inputs directly into the decoder input stream.Parameters
12B
Context
128K
KV / token
328 KiB
Fact sheet
- Scale
- 12B parameters
- Context window
- 128,000 tokens
- Decoder type
- Dense
- Attention
- GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- Layer mix
- 40 sliding-window + 8 global
- KV cache / token (bf16)
- 328 KiB · Very high
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source9.4
- Scientific
- 3.2
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Google, released 2025-03-11. Dense decoder, 27B parameters, 128K context. Attention: GQA with QK-Norm and 5:1 sliding-window/global attention.
Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
Google, released 2026-04-02. Dense decoder, 31B parameters, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.