Cisco, released 2026-07-21. Dense decoder, 1B parameters, 128K context. Attention: GQA with RoPE.
DenseMeta 🇺🇸 · September 25, 2024
Llama 3.2 (1B)
Small dense Llama baseline in the Qwen comparison, with fewer layers but more width.
Key detail
Wider architecture with more heads than Qwen3 0.6B.Parameters
1B
Context
128K
KV / token
32 KiB
Fact sheet
- Scale
- 1B parameters
- Context window
- 128,000 tokens
- Decoder type
- Dense
- Attention
- GQA
- Layer mix
- 16 GQA
- KV cache / token (bf16)
- 32 KiB · Low
- License
- Llama Community License Agreement (variant-specific)
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source4.8
- Scientific
- 2.7
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Alibaba Cloud, released 2025-04-28. Dense decoder, 600M parameters, 32K context. Attention: GQA.
Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.
Meta, released 2024-09-25. Dense decoder, 3B parameters, 128K context. Attention: GQA.