Skip to content

Positions & normalization · NoPE

No Positional Embeddings

Some or all layers drop explicit positional encodings and rely on the causal mask.

Because a decoder's causal mask already prevents tokens from seeing the future, the model can infer order implicitly. NoPE layers omit RoPE entirely.

Studies suggest this can help length generalisation. SmolLM3 removes RoPE from every fourth layer; Kimi K3 goes NoPE throughout.

Read the full explainer Browse all 18 models
Annotated NoPE figure about length generalization. Figure from Sebastian Raschka’s explainer.

Adoption over time

Models in the atlas using NoPE, per half-year of release.

Loading

Used by

18

models in the atlas

Earliest: SmolLM3 (3B) (Jun 19, 2025)

Often paired with

Models using NoPE

  • Architecture diagram of Mistral Large 4 (1T)

    Mistral Large 4 (1T)

    Mistral AI, released 2026-10-29. Sparse MoE decoder, 1T total, 52B active, 1M context. Attention: MLA with modified DeepSeek Sparse Attention, index reuse, and KV sharing.

  • Architecture diagram of Recursive Flint (40B-A2.5B)

    Recursive Flint (40B-A2.5B)

    Recursive, released 2026-10-18. Sparse MoE decoder, 40B total, 2.5B active, 256K context. Attention: GQA with QK-Norm, learned attention sinks, and sliding-window/global attention.

  • Architecture diagram of Reflection Beam (501B)

    Reflection Beam (501B)

    Reflection, released 2026-10-16. Sparse MoE decoder, 501B total, 23B active, 1M context. Attention: Gated GQA with QK-Norm and sliding-window/global attention.

  • Architecture diagram of Muse Spark Open (816B-A42B)

    Muse Spark Open (816B-A42B)

    Meta, released 2026-10-15. Sparse MoE decoder, 816B total, 42B active, 1M context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.

  • Architecture diagram of Kolibri-1 (78B)

    Kolibri-1 (78B)

    Aleph Alpha, released 2026-10-03. Sparse MoE decoder, 78B total, 3.5B active, 256K context. Attention: GQA with QK-Norm and 4:1 sliding-window/global attention.

  • Architecture diagram of Ember-1 (2.8T)

    Ember-1 (2.8T)

    Fireworks AI, released 2026-09-23. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.

  • Architecture diagram of GLM-5.3-Flash (320B)

    GLM-5.3-Flash (320B)

    Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.

  • Architecture diagram of Muse Glimmer (30B)

    Muse Glimmer (30B)

    Meta, released 2026-08-10. Dense decoder, 30B parameters, 128K context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.

  • Architecture diagram of Kimi K3 (2.8T)

    Kimi K3 (2.8T)

    Moonshot AI, released 2026-07-27. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.

  • Architecture diagram of Solar Open 2 (250B)

    Solar Open 2 (250B)

    Upstage, released 2026-07-22. Hybrid MoE decoder, 250B total, 15B active, 1M context. Attention: 3:1 Kimi Delta Attention and gated GQA with NoPE.

  • Architecture diagram of Soofi S (30B-A3B)

    Soofi S (30B-A3B)

    SOOFI Consortium, released 2026-07-10. Hybrid MoE decoder, 32B total, 3.2B active, 1M context. Attention: Mostly Mamba-2 with 6 GQA layers and no positional embeddings (NoPE).

  • Architecture diagram of North Mini Code (30B-A3B)

    North Mini Code (30B-A3B)

    Cohere, released 2026-06-05. Sparse MoE decoder, 30B total, 3B active, 256K context. Attention: 8:1 GQA with 3:1 sliding-window/global attention.

See all 18 models

Search the atlas

Jump to a model, concept or page