# Motif 3 Beta (314B)

> Large sparse MoE that combines Grouped Differential Latent Attention, per-expert PolyNorm, and modified mHC in a 53-layer decoder.

| Property | Value |
| --- | --- |
| Lab | Motif Technologies (South Korea) |
| Released | 2026-07-21 |
| Decoder type | Sparse MoE |
| Family | Sparse MoE |
| Scale | 314B total, 13B active (4.1% active) |
| Context window | 262,144 tokens |
| Attention | GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention |
| Layer mix | 39 sliding-window + 14 full GDLA |
| KV cache / token (bf16) | 530 KiB (Very high) |
| License | Motif Technologies Non-Commercial License |

**Key detail:** Routes each token through 8 of 384 experts plus one shared expert, with per-expert PolyNorm and a 3:1 sliding-window/full GDLA pattern.

## Architecture concepts

- [Multi-Head Latent Attention (MLA)](https://llm-atlas-7or.pages.dev/concepts/mla.md): Keys and values are compressed into a small latent vector that is cached instead of full K/V.
- [Sliding Window Attention (SWA)](https://llm-atlas-7or.pages.dev/concepts/swa.md): Some layers only attend to a local window of recent tokens instead of the whole context.
- [Manifold-constrained Hyper-Connections (mHC)](https://llm-atlas-7or.pages.dev/concepts/mhc.md): Several parallel residual streams with learned, constrained mixing between them.
- [Mixture of Experts (MoE)](https://llm-atlas-7or.pages.dev/concepts/moe.md): Many feed-forward experts, but a router sends each token to only a few of them.
- [PolyNorm (PolyNorm)](https://llm-atlas-7or.pages.dev/concepts/polynorm.md): A learned activation mixing normalised linear, quadratic and cubic terms.
- [Multi-Token Prediction (MTP)](https://llm-atlas-7or.pages.dev/concepts/mtp.md): Extra heads learn to predict several future tokens, and can draft tokens at inference.

## Benchmark (Artificial Analysis Intelligence Index)

- AA Intelligence Index v4.3.2: total 32.3 (Scientific 23.2)

## Links

- Architecture diagram: https://sebastianraschka.com/llm-architecture-gallery/images/architectures/motif-3-beta.webp
- config.json: https://huggingface.co/Motif-Technologies/Motif-3-Beta/blob/main/config.json
- License: https://huggingface.co/Motif-Technologies/Motif-3-Beta/blob/main/README.md#license
- Source card (source of truth): https://sebastianraschka.com/llm-architecture-gallery/#card-motif-3-beta
- Web page: https://llm-atlas-7or.pages.dev/models/motif-3-beta/
- JSON: https://llm-atlas-7or.pages.dev/api/models/motif-3-beta.json

_Data from Sebastian Raschka's LLM Architecture Gallery (https://sebastianraschka.com/llm-architecture-gallery/). Synced 2026-11-04._