# GPT-2 XL (1.5B)

> Late-2019 dense baseline included here as a reference point for how much decoder stacks have changed since GPT-2.

| Property | Value |
| --- | --- |
| Lab | OpenAI (United States) |
| Released | 2019-11-05 |
| Decoder type | Dense |
| Family | Dense |
| Scale | 1.5B parameters |
| Context window | 1,024 tokens |
| Attention | MHA with learned absolute positional embeddings |
| Layer mix | 48 MHA |
| KV cache / token (bf16) | 300 KiB (High) |
| License | OpenAI "Modified MIT" license |

**Key detail:** Classic GPT-2 recipe with dropout, GELU, LayerNorm, and full multi-head attention.

## Architecture concepts

- [Multi-Head Attention (MHA)](https://llm-atlas-7or.pages.dev/concepts/mha.md): The original transformer attention: every head has its own queries, keys and values.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/gpt-2-xl.webp
- View in article: https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzing-the#%C2%A72-coming-from-gpt-2
- config.json: https://huggingface.co/openai-community/gpt2-xl/blob/main/config.json
- Tech report: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-gpt-2-xl-1-5b
- Web page: https://llm-atlas-7or.pages.dev/models/gpt-2-xl-1-5b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/gpt-2-xl-1-5b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._