Sparse MoEMotif Technologies 🇰🇷 · July 21, 2026

Motif 3 Beta (314B)

Large sparse MoE that combines Grouped Differential Latent Attention, per-expert PolyNorm, and modified mHC in a 53-layer decoder.

Key detail

Routes each token through 8 of 384 experts plus one shared expert, with per-expert PolyNorm and a 3:1 sliding-window/full GDLA pattern.

Total params

314B

Active / token

13B

Context

256K

Fact sheet

Scale: 314B total, 13B active (4.1% active)
Context window: 262,144 tokens
Decoder type: Sparse MoE
Attention: GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention
Layer mix: 39 sliding-window + 14 full GDLA
KV cache / token (bf16): 530 KiB · Very high
License: Motif Technologies Non-Commercial License
Released: July 21, 2026

Parameters active per token

4.1%

13B of 314B weights run for each token.

KV cache per token (bf16)

530 KiB

Smaller than 2% of models in the atlas. How it’s calculated

32.3

Use the ← and → keys to step through releases.

DeepSeek V4-Flash (284B)
DeepSeek, released 2026-04-24. Sparse MoE decoder, 284B total, 13B active, 1M context. Attention: MLA-style CSA/HCA with mHC.
Tencent Hy4-preview (770B-A49B)
Tencent, released 2026-08-28. Sparse MoE decoder, 770B total, 49B active, 1M context. Attention: Gated MLA with DeepSeek Sparse Attention and IndexCache.
DeepSeek V4-Pro (1.6T)
DeepSeek, released 2026-04-24. Sparse MoE decoder, 1.6T total, 49B active, 1M context. Attention: MLA-style CSA/HCA with mHC.
GLM-5.3-Flash (320B)
Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.