Sparse MoEThinking Machines Lab 🇺🇸 · July 15, 2026

Inkling (975B)

Thinking Machines Lab's native multimodal model combines a 66-layer sparse MoE decoder with a 5:1 mix of local and global GQA.

Key detail

Uses learned relative-position bias instead of RoPE and adds four short convolutions to each decoder layer.

Total params

975B

Active / token

41B

Context

Fact sheet

Scale: 975B total, 41B active (4.2% active)
Context window: 1,048,576 tokens
Decoder type: Sparse MoE
Attention: GQA with QK-Norm and learned relative-position bias in a 5:1 sliding-window/global pattern
Layer mix: 55 sliding-window + 11 global GQA
KV cache / token (bf16): 484 KiB · Very high
License: Apache License 2.0
Released: July 15, 2026

Parameters active per token

4.2%

41B of 975B weights run for each token.

KV cache per token (bf16)

484 KiB

Smaller than 5% of models in the atlas. How it’s calculated

12.5

Use the ← and → keys to step through releases.

Step-5 Preview (600B)
StepFun, released 2026-09-20. Sparse MoE decoder, 600B total, 27B active, 1M context. Attention: Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention.
MiniMax-M2.5 (230B)
MiniMax, released 2026-02-12. Sparse MoE decoder, 230B total, 10B active, 192K context. Attention: GQA with QK-Norm.
Xiaomi MiMo-V2.6-Pro-RL (1.02T)
Xiaomi, released 2026-09-21. Sparse MoE decoder, 1.02T total, 42B active, 1M context. Attention: GQA with 6:1 sliding-window/global attention.
Muse Spark Open (816B-A42B)
Meta, released 2026-10-15. Sparse MoE decoder, 816B total, 42B active, 1M context. Attention: Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention.