Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
DenseGoogle 🇺🇸 · March 11, 2025
Gemma 3 (27B)
Gemma's flagship text stack leans on local attention more aggressively than Gemma 2.
Key detail
Built around a 27B sweet spot with heavier local attention and a large 262k multilingual vocabulary.Parameters
27B
Context
128K
KV / token
496 KiB
Fact sheet
- Scale
- 27B parameters
- Context window
- 128,000 tokens
- Decoder type
- Dense
- Attention
- GQA with QK-Norm and 5:1 sliding-window/global attention
- Layer mix
- 52 sliding-window + 10 global
- KV cache / token (bf16)
- 496 KiB · Very high
- License
- Gemma Terms of Use + Gemma Prohibited Use Policy
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source4.9
- General
- 6.9
- Scientific
- 2.2
- Coding
- 11.6
- Agents
- 0
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Google, released 2026-04-02. Dense decoder, 31B parameters, 256K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
Ai2, released 2025-11-20. Dense decoder, 32B parameters, 64K context. Attention: GQA with QK-Norm and 3:1 sliding-window/global attention.
Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.