Z.ai, released 2025-07-28. Sparse MoE decoder, 355B total, 32B active, 128K context. Attention: GQA with QK-Norm.
Sparse MoETencent 🇨🇳 · April 23, 2026
Tencent Hy3-preview (295B-A21B)
Tencent Hunyuan's 295B sparse MoE preview combines a dense first FFN layer, GQA with QK-Norm, and one extra MTP layer for faster generation.
Key detail
Uses 192 routed experts, 8 routed plus 1 shared expert active per token, sigmoid routing with expert bias, and a 3.8B MTP layer.Total params
295B
Active / token
21B
Context
256K
Fact sheet
- Scale
- 295B total, 21B active (7.1% active)
- Context window
- 262,144 tokens
- Decoder type
- Sparse MoE
- Attention
- GQA with QK-Norm
- Layer mix
- 80 GQA + 1 MTP layer
- KV cache / token (bf16)
- 320 KiB · Very high
- License
- Tencent Hy Community License Agreement
- Released
Efficiency
Parameters active per token
7.1%
21B of 295B weights run for each token.
AA Intelligence Index v4.3.2
Source17
- Scientific
- 3.6
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
Z.ai, released 2025-12-22. Sparse MoE decoder, 355B total, 32B active, 198K context. Attention: GQA with QK-Norm.
MiniMax, released 2025-10-23. Sparse MoE decoder, 230B total, 10B active, 192K context. Attention: GQA with QK-Norm and partial RoPE.
MiniMax, released 2026-02-12. Sparse MoE decoder, 230B total, 10B active, 192K context. Attention: GQA with QK-Norm.