Skip to content

Efficiency & sharing · PLE

Per-Layer Embeddings

Each layer gets its own token-specific embedding signal from a large lookup table.

Lookup tables are cheap to run and can live outside fast memory, so small models can store extra token-specific information per layer without making the transformer stack itself more expensive.

It is why Gemma 4 E2B is described as 2.3B effective parameters but 5.1B including embeddings. The “E” stands for effective.

Read the full explainer Browse all 3 models
Simplified Gemma 4 block with the per-layer embedding residual path. Figure from Sebastian Raschka’s explainer.

Adoption over time

Models in the atlas using PLE, per half-year of release.

Loading

Used by

3

models in the atlas

Earliest: Gemma 4 (E4B) (Apr 2, 2026)

Often paired with

Models using PLE

  • Architecture diagram of Qwen3.8-Flash-Next (125B-A6B)

    Qwen3.8-Flash-Next (125B-A6B)

    Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).

  • Architecture diagram of Gemma 4 (E2B)

    Gemma 4 (E2B)

    Google, released 2026-04-02. Dense decoder, 5.1B parameters, 128K context. Attention: Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention.

  • Architecture diagram of Gemma 4 (E4B)

    Gemma 4 (E4B)

    Google, released 2026-04-02. Dense decoder, 8B parameters, 128K context. Attention: GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention.

Search the atlas

Jump to a model, concept or page