# PolyNorm (PolyNorm)

> A learned activation mixing normalised linear, quadratic and cubic terms.

Category: Feed-forward & experts

Feed-forward blocks moved from GELU to SwiGLU-style gating with a fixed SiLU. PolyNorm replaces that fixed function with a learned combination of separately RMS-normalised powers of the pre-activation.

Motif 3 Beta uses a grouped PolyNorm inside each expert.

**Trade-off:** A more flexible nonlinearity for a little extra compute and tuning.

## Models using PolyNorm (1)

- [Motif 3 Beta (314B)](https://llm-atlas-7or.pages.dev/models/motif-3-beta.md) — Motif Technologies, 2026-07-21

Full explainer with figures by Sebastian Raschka: https://sebastianraschka.com/llm-architecture-gallery/polynorm/
Web page: https://llm-atlas-7or.pages.dev/concepts/polynorm/