DenseGoogle 🇺🇸 · August 14, 2025

Gemma 3 (270M)

Tiny Gemma 3 variant that preserves the family's local-global attention recipe at a toy scale.

Key detail

Keeps the Gemma 3 stack shape while shrinking down to 4 attention heads, a single KV head, and the same 262k vocabulary.

Parameters

270M

Context

128K

KV / token

18 KiB

Fact sheet

Scale: 270M parameters
Context window: 128,000 tokens
Decoder type: Dense
Attention: Multi-query attention with QK-Norm and 5:1 sliding-window/global attention
Layer mix: 15 sliding-window + 3 global
KV cache / token (bf16): 18 KiB · Very low
License: Gemma Terms of Use + Gemma Prohibited Use Policy
Released: August 14, 2025

Parameters active per token

100%

Dense: every weight runs for every token.

KV cache per token (bf16)

18 KiB

Smaller than 83% of models in the atlas. How it’s calculated

5.1

Use the ← and → keys to step through releases.

Dory1.0 (4B)
NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.
Gemma 4 (E2B)
Google, released 2026-04-02. Dense decoder, 5.1B parameters, 128K context. Attention: Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention.
OLMo 3 (7B)
Ai2, released 2025-11-20. Dense decoder, 7B parameters, 64K context. Attention: MHA with QK-Norm and 3:1 sliding-window/global attention.
Gemma 4 (E4B)
Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.