Google, released 2026-06-03. Dense decoder, 12B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.
DenseMicrosoft 🇺🇸 · December 12, 2024
Phi-4 (14B)
Microsoft's 14B dense Phi refresh stays close to Phi-3-medium but swaps its sliding-window attention for full-context GQA and a larger tokenizer.
Key detail
Classic pre-norm RMSNorm stack with GQA, 40 heads, 10 KV heads, and a 100,352-token vocabulary.Parameters
14B
Context
16K
KV / token
200 KiB
Fact sheet
- Scale
- 14B parameters
- Context window
- 16,384 tokens
- Decoder type
- Dense
- Attention
- GQA with RoPE
- Layer mix
- 40 GQA
- KV cache / token (bf16)
- 200 KiB · High
- License
- MIT License
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source5.9
- Scientific
- 1.9
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Mistral AI, released 2025-03-18. Dense decoder, 24B parameters, 128K context. Attention: Standard GQA.
Meta, released 2024-04-18. Dense decoder, 8B parameters, 8K context. Attention: GQA with RoPE.
Alibaba Cloud, released 2025-04-28. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm.