DeepSeek, released 2026-04-24. Sparse MoE decoder, 1.6T total, 49B active, 1M context. Attention: MLA-style CSA/HCA with mHC.
Sparse MoETencent 🇨🇳 · August 28, 2026
Tencent Hy4-preview (770B-A49B)
Tencent's 770B sparse MoE combines gated DeepSeek Sparse Attention, cross-layer index reuse, and four-stream identity Hyper-Connections.
Key detail
Uses 256 routed experts with 8 routed plus 1 shared expert active per token, a top-2,048 DSA indexer, cross-layer index reuse, and four iHC residual streams.Total params
770B
Active / token
49B
Context
1M
Fact sheet
- Scale
- 770B backbone, 49B active (6.4% active); plus 10B MTP
- Context window
- 1,048,576 tokens
- Decoder type
- Sparse MoE
- Attention
- Gated MLA with DeepSeek Sparse Attention and IndexCache
- Layer mix
- 78 gated MLA/DSA + 1 MTP layer
- KV cache / token (bf16)
- 87.8 KiB · Moderate
- License
- Apache License 2.0
- Released
Efficiency
Parameters active per token
6.4%
49B of 770B weights run for each token.
Architecture concepts
Sources & resources
Use the ← and → keys to step through releases.
Similar architectures
DeepSeek, released 2026-04-24. Sparse MoE decoder, 284B total, 13B active, 1M context. Attention: MLA-style CSA/HCA with mHC.
Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.
Z.ai, released 2026-02-11. Sparse MoE decoder, 744B total, 40B active, 198K context. Attention: MLA with DeepSeek Sparse Attention.