Zyphra, released 2026-05-06. Sparse MoE decoder, 8.4B total, 760M active, 128K context. Attention: CCA with 4:1 GQA, RoPE, and Q/K L2 norm.
Sparse & compressed attention · CCA
Compressed Convolutional Attention
Attention computed directly in a compressed latent space, with convolutional mixing.
Where MLA uses a latent mainly as a compact cache format and decompresses it for attention, CCA performs the attention operation itself in the compressed space and adds convolutions to mix information.
ZAYA1-8B by Zyphra uses CCA together with grouped-query attention.
Adoption over time
Models in the atlas using CCA, per half-year of release.
Often paired with