Hybrid MoENVIDIA 🇺🇸 · October 15, 2026

Nemotron 3.5 Super VL (120B-A12B)

NVIDIA's vision-language model extends the Nemotron 3 Super backbone with C-RADIO vision encoding and shared-weight multi-token prediction. The figure shows the language backbone, with the vision encoder and MTP omitted.

Key detail

Routes each token to 22 of 512 experts in a 1,024-dimensional latent space, alongside one shared expert at the full 4,096-dimensional width.

Total params

120B

Active / token

12B

Context

Fact sheet

Scale: 120B total, 12B active (10% active)
Context window: 1,048,576 tokens
Decoder type: Hybrid MoE
Attention: Mostly Mamba-2 with eight GQA layers
Layer mix: 8 GQA + 40 Mamba-2 + 40 LatentMoE; GQA + MoE MTP predictor
KV cache / token (bf16): 8 KiB · Very low
License: OpenMDW 1.1
Released: October 15, 2026

Parameters active per token

10%

12B of 120B weights run for each token.

KV cache per token (bf16)

8 KiB

Smaller than 91% of models in the atlas. How it’s calculated

AA Intelligence Index v5.0

Source

13.6

Knowledge Work: 19.5
Coding: 3.7
Science: 5.4
General: 25.9

Architecture concepts

Sources & resources

config.json License Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

Nemotron 3 Super (120B-A12B)
NVIDIA, released 2026-03-11. Hybrid MoE decoder, 120B total, 12B active, 1M context. Attention: Mostly Mamba-2 with a few GQA layers.
Qwen3.8-Flash-Next (125B-A6B)
Alibaba Cloud, released 2026-08-26. Hybrid MoE decoder, 125B total, 6B active, 256K context. Attention: 3:1 Gated DeltaNet and Qwen Sparse Attention (QSA).
Ornith 1.5 (35B-A3B)
Ornith, released 2026-08-23. Hybrid MoE decoder, 35B total, 3B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.
Nemotron 3.5 Lightning (30B-A3B)
NVIDIA, released 2026-08-11. Hybrid MoE decoder, 30B total, 3B active, 1M context. Attention: Mostly Mamba-2 with six GQA layers.