Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.
DenseCohere 🇨🇦 · February 13, 2026
Tiny Aya (3.35B)
Compact multilingual model from Cohere with a rare parallel transformer block.
Key detail
Runs attention and the MLP in parallel while mixing RoPE with NoPE.Parameters
3.4B
Context
8K
KV / token
72 KiB
Fact sheet
- Scale
- 3.35B parameters
- Context window
- 8,192 tokens
- Decoder type
- Dense
- Attention
- GQA with 3:1 sliding-window attention
- Layer mix
- 27 sliding-window + 9 global
- KV cache / token (bf16)
- 72 KiB · Low
- License
- Creative Commons Attribution-NonCommercial 4.0
- Released
Efficiency
Parameters active per token
100%
Dense: every weight runs for every token.
AA Intelligence Index v4.3.2
Source4.8
- Scientific
- 2.6
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Hugging Face, released 2025-06-19. Dense decoder, 3B parameters, 128K context. Attention: GQA with periodic NoPE layers.
NVIDIA, released 2026-10-16. Dense decoder, 4B parameters, 256K context. Attention: Gated GQA; 3:1 sliding-window/full, RoPE.
Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.