Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
DenseGoogle 🇺🇸 · April 2, 2026
Gemma 4 (E2B)
Smallest Gemma 4 edge model keeps the family's hybrid attention stack and adds native audio on a phone-scale multimodal footprint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 2.3B than a full 5.1B dense model.
Key detail
Uses a double-wide GELU MLP plus a single KV head to stay light enough for offline edge deployments.Parameters
5.1B
Context
128K
KV / token
35 KiB
Fact sheet
- Scale
- 5.1B parameters (2.3B effective)
- Context window
- 128,000 tokens
- Decoder type
- Dense
- Attention
- Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention
- Layer mix
- 28 sliding-window + 7 global
- KV cache / token (bf16)
- 35 KiB · Low
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source6.5
- Scientific
- 2.4
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.
Ai2, released 2025-11-20. Dense decoder, 7B parameters, 64K context. Attention: MHA with QK-Norm and 3:1 sliding-window/global attention.
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.