Google, released 2026-04-02. Dense decoder, 5.1B parameters, 128K context. Attention: Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention.
DenseGoogle 🇺🇸 · April 2, 2026
Gemma 4 (E4B)
Larger Gemma 4 edge variant keeps the same multimodal hybrid recipe but doubles width and KV heads for a stronger 128K mobile checkpoint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 4.5B than a full 8B dense model.
Key detail
Steps up to a 42-layer stack with 2 KV heads while keeping the same edge-oriented local/global template.Parameters
8B
Context
128K
KV / token
84 KiB
Fact sheet
- Scale
- 8B parameters (4.5B effective)
- Context window
- 128,000 tokens
- Decoder type
- Dense
- Attention
- GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- Layer mix
- 35 sliding-window + 7 global
- KV cache / token (bf16)
- 84 KiB · Moderate
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source7.5
- Scientific
- 2.6
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.
Google, released 2025-03-11. Dense decoder, 27B parameters, 128K context. Attention: GQA with QK-Norm and 5:1 sliding-window/global attention.