# Tencent Hy4-preview (770B-A49B)

> Tencent's 770B sparse MoE combines gated DeepSeek Sparse Attention, cross-layer index reuse, and four-stream identity Hyper-Connections.

| Property | Value |
| --- | --- |
| Lab | Tencent (China) |
| Released | 2026-08-28 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 770B backbone, 49B active (6.4% active); plus 10B MTP |
| Context window | 1,048,576 tokens |
| Attention | Gated MLA with DeepSeek Sparse Attention and IndexCache |
| Layer mix | 78 gated MLA/DSA + 1 MTP layer |
| KV cache / token (bf16) | 87.8 KiB (Moderate) |
| License | Apache License 2.0 |

**Key detail:** Uses 256 routed experts with 8 routed plus 1 shared expert active per token, a top-2,048 DSA indexer, cross-layer index reuse, and four iHC residual streams.

## Architecture concepts

- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [DeepSeek Sparse Attention (DSA)](https://llm-atlas-7or.pages.dev/concepts/deepseek-sparse-attention.md): A lightweight indexer scores past tokens and each query attends only to its top-k picks.
- [IndexShare (IndexShare)](https://llm-atlas-7or.pages.dev/concepts/indexshare.md): Nearby layers reuse the token selection computed by one DSA indexer.
- [Gated Attention (Gated attention)](https://llm-atlas-7or.pages.dev/concepts/gated-attention.md): A learned sigmoid gate scales the attention output before it re-enters the residual stream.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/tencent-hy4.webp
- config.json: https://huggingface.co/tencent/Hy4-preview/blob/main/config.json
- License: https://huggingface.co/tencent/Hy4-preview/blob/main/LICENSE
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-tencent-hy4-preview-770b-a49b
- Web page: https://llm-atlas-7or.pages.dev/models/tencent-hy4-preview-770b-a49b/
- JSON: https://llm-atlas-7or.pages.dev/api/models/tencent-hy4-preview-770b-a49b.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._