Sparse hybridUpstage 🇰🇷 · July 22, 2026

Solar Open 2 (250B)

Upstage's sparse hybrid alternates one gated GQA layer with three Kimi Delta Attention layers and reaches a 1M-token context without positional embeddings.

Key detail

Uses a repeating gated-GQA/KDA hybrid stack, kernel-4 ShortConv in KDA, and top-8 routing over 320 experts plus one shared expert.

Total params

250B

Active / token

15B

Context

Fact sheet

Scale: 250B total, 15B active (6% active)
Context window: 1,048,576 tokens
Decoder type: Sparse hybrid
Attention: 3:1 Kimi Delta Attention and gated GQA with NoPE
Layer mix: 12 gated GQA + 36 Kimi Delta Attention
KV cache / token (bf16): 48 KiB · Low
License: Upstage Solar License
Released: July 22, 2026

Parameters active per token

6.0%

15B of 250B weights run for each token.

KV cache per token (bf16)

48 KiB

Smaller than 67% of models in the atlas. How it’s calculated

AA Intelligence Index v4.3.2

Source

24.7

Scientific: 17.1

Architecture concepts

Sources & resources

config.json License Tech report Original gallery card

Compare with… JSON

Use the ← and → keys to step through releases.

Similar architectures

Ornith 1.5 (35B-A3B)
Ornith, released 2026-08-23. Hybrid MoE decoder, 35B total, 3B active, 256K context. Attention: 3:1 Gated DeltaNet and Gated Attention.
Kimi K3 (2.8T)
Moonshot AI, released 2026-07-27. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.
Ember-1 (2.8T)
Fireworks AI, released 2026-09-23. Hybrid MoE decoder, 2.8T total, 104B active, 1M context. Attention: 3:1 Kimi Delta Attention and Gated MLA with NoPE.
GLM-5.3-Flash (320B)
Z.ai, released 2026-08-26. Hybrid MoE decoder, 320B total, 18B active, 1M context. Attention: 3:1 Kimi Delta Attention and MLA with DeepSeek Sparse Attention.