The DeepSeekMoE Architecture.

Everything so far — routing, distributed specialisation, balancing, capacity — was setup. DeepSeekMoE is the synthesis: two architectural changes to the classic MoE block that take the messy empirical reality of 5.3 and design around it instead of fighting it. Those two ideas — fine-grained expert segmentation and shared expert isolation — are why DeepSeek's MoE works at 671B parameters with only 37B active.

Where conventional MoE leaves value on the table

A classic MoE block (GShard, Switch) has N experts, each a full-size FFN, and routes each token to top-k (often k=1 or 2). Two inefficiencies, both visible in 5.3's findings:

• Knowledge redundancy. Every routed expert ends up re-learning the same common patterns (syntax, frequent tokens) because every token needs them regardless of which expert it lands on. Capacity is wasted storing the same basics in many experts.
• Coarse routing. With few large experts, the router's choice is blunt. Real specialisation is distributed and combinatorial (5.3), but top-2-of-8 gives only 28 possible expert combinations — far too coarse to express fine distinctions.

Idea 1 — Fine-grained expert segmentation

Split each expert into m smaller ones. Take the N experts of hidden size d_ff and replace them with mN experts of hidden size d_ff/m, while routing to mk of them. Total parameters and active FLOPs are unchanged — you just sliced the same compute into finer pieces — but the routing flexibility explodes.

conventional:   N experts,  top-k
fine-grained:   mN experts, top-(mk),  each expert 1/m the size

# active params identical:  mk * (d_ff/m) = k * d_ff
# but the number of expert COMBINATIONS the router can form:

conventional combinations:  C(N, k)      e.g. C(16, 2)   = 120
fine-grained  combinations:  C(mN, mk)    e.g. C(64, 8)   = 4,426,165,368

Same compute, ~37 million× more ways to combine experts. The router gets a vastly finer palette — exactly what 5.3 said it needs, since specialisation is distributed and combinatorial rather than one-expert-per-topic. Fine-grained experts let the model assemble a bespoke combination per token from many small specialists.

Idea 2 — Shared expert isolation

Carve off K_s experts that are always on — every token goes through them, no routing decision. The remaining experts are routed as usual. The shared experts absorb the common knowledge that 5.3 showed gets smeared across everything; the routed experts are then freed to capture only the distinctions that actually differ between tokens.

output = sum over shared experts s of   FFN_s(x)              # always applied
       + sum over routed top-k experts r of  g_r * FFN_r(x)   # gated, selected
       + x                                                    # residual

This directly removes the redundancy problem: common patterns are learned once, in the shared experts, instead of redundantly in every routed expert. Each routed expert now carries less duplicated baseline and more genuine specialisation — making the (still distributed) specialisation more meaningful per parameter.

The configurations, concretely

The same recipe scaled across two model generations:

	DeepSeek-V2	DeepSeek-V3
total parameters	236 B	671 B
active per token	21 B	37 B
shared experts (Ks)	2	1
routed experts (N)	160	256
top-k routed	6	8
experts per token	2 + 6 = 8	1 + 8 = 9
router gate	softmax	sigmoid
balancing	aux losses (×3)	aux-loss-free bias (5.4)
routing locality	device-limited	node-limited (≤ 4 nodes)
token dropping	minimal	none

Note how small the active fraction is: V3 activates 37 / 671 ≈ 5.5% of parameters per token. You pay 37B-model inference cost while the model knows 671B worth — that's the whole MoE bargain (5.1), pushed hard by fine-grained experts.

Why each choice follows from the earlier articles

• Many fine-grained routed experts (256). Follows from 5.3: specialisation is distributed, so give the router a fine, combinatorial palette rather than a few blunt experts.
• Shared experts. Follows from 5.3's redundancy finding: factor out the common knowledge once instead of duplicating it across routed experts.
• Sigmoid gating (V3). With 256 experts, a softmax over all of them is numerically awkward and couples the scores; per-expert sigmoid affinities decouple them and pair cleanly with the additive selection bias (5.4).
• Aux-loss-free bias balancing. Follows from 5.4: get balance without the gradient-interference quality tax.
• Node-limited routing. Follows from 5.5: cap each token's experts to ≤4 nodes to bound all-to-all traffic and keep load satisfiable without dropping.

DeepSeekMoE isn't one clever trick — it's a stack of choices that each answer a specific failure mode from the previous articles. Fine-grained experts for the routing flexibility, shared experts for the redundancy, sigmoid + bias for clean balanced selection, node-limited routing for the network. Together they make a 671B model train and serve at 37B cost.

What it buys, measured

DeepSeek's ablations show fine-grained + shared experts beating a conventional MoE of equal total and active parameters — the gains are architectural, not just from scale. And the engineering payoff is stark: DeepSeek-V3 trained for a small fraction of the cost of comparable dense frontier models precisely because only ~5.5% of parameters activate per token, while quality tracks much larger dense models.

The shared-expert caveat

Shared experts are always on, so they're always in the active-parameter budget and on the critical path for every token. Make them too large and you erode the sparsity advantage; too small and they can't hold the common knowledge. DeepSeek-V3's choice of a single shared expert (down from V2's two) reflects tuning this balance as the routed pool grew to 256 — more fine-grained routed capacity meant less needed to be carried in the always-on path.

Next, 5.7 turns this into code: a minimal but faithful DeepSeekMoE block in PyTorch — fine-grained routed experts, shared experts, sigmoid gating, top-k selection with the balancing bias, and the gather/scatter that makes it run.

References

• Dai et al. (2024), DeepSeekMoE: Towards Ultimate Expert Specialization — fine-grained segmentation and shared-expert isolation. arXiv:2401.06066
• DeepSeek-AI (2024), DeepSeek-V2 — 236B/21B, 2 shared + 160 routed, device-limited routing. arXiv:2405.04434
• DeepSeek-AI (2024), DeepSeek-V3 — 671B/37B, sigmoid gating, node-limited routing, drop-free. arXiv:2412.19437
• Fedus et al. (2021), Switch Transformers — the conventional MoE baseline being improved on. arXiv:2101.03961