Skip to content

Feed-forward & experts · PolyNorm

PolyNorm

A learned activation mixing normalised linear, quadratic and cubic terms.

Feed-forward blocks moved from GELU to SwiGLU-style gating with a fixed SiLU. PolyNorm replaces that fixed function with a learned combination of separately RMS-normalised powers of the pre-activation.

Motif 3 Beta uses a grouped PolyNorm inside each expert.

Read the full explainer Browse all 1 models
SwiGLU compared with PolyNorm. Figure from Sebastian Raschka’s explainer.

Adoption over time

Models in the atlas using PolyNorm, per half-year of release.

Loading

Used by

1

models in the atlas

Earliest: Motif 3 Beta (314B) (Jul 21, 2026)

Often paired with

Models using PolyNorm

  • Architecture diagram of Motif 3 Beta (314B)

    Motif 3 Beta (314B)

    Motif Technologies, released 2026-07-21. Sparse MoE decoder, 314B total, 13B active, 256K context. Attention: GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention.

Search the atlas

Jump to a model, concept or page