ByteDance, released 2025-10-29. Dense decoder, 2.6B parameters, 64K context. Attention: MHA with RoPE.
DenseOpenAI 🇺🇸 · November 5, 2019
GPT-2 XL (1.5B)
Late-2019 dense baseline included here as a reference point for how much decoder stacks have changed since GPT-2.
Key detail
Classic GPT-2 recipe with dropout, GELU, LayerNorm, and full multi-head attention.Parameters
1.5B
Context
1K
KV / token
300 KiB
Fact sheet
- Scale
- 1.5B parameters
- Context window
- 1,024 tokens
- Decoder type
- Dense
- Attention
- MHA with learned absolute positional embeddings
- Layer mix
- 48 MHA
- KV cache / token (bf16)
- 300 KiB · High
- License
- OpenAI "Modified MIT" license
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Ai2, released 2024-11-25. Dense decoder, 7B parameters, 4K context. Attention: MHA with QK-Norm.
Ai2, released 2025-11-20. Dense decoder, 7B parameters, 64K context. Attention: MHA with QK-Norm and 3:1 sliding-window/global attention.
Meta, released 2024-09-25. Dense decoder, 1B parameters, 128K context. Attention: GQA.