Introduction to Mixture of Experts.

Phases 1–4 were about attention — how a Transformer mixes information across positions, and how MLA and RoPE make that cheap. But attention is only half of a Transformer block. The other half is the feed-forward network (FFN), and in large models the FFN is where most of the parameters and most of the compute live. Phase 5 is about the idea that lets DeepSeek scale that half to hundreds of billions of parameters while only paying for a small slice per token: Mixture of Experts.

This article sets up the problem MoE solves, the core "sparse activation" idea, and the vocabulary the rest of Phase 5 will use. No prior MoE knowledge assumed.

Where the parameters actually go

A standard Transformer block has two sublayers: multi-head attention, then a position-wise FFN. The FFN is a simple two-layer MLP applied to each token independently:

FFN(x) = W_2 · activation(W_1 · x)

where for a model of width d_model:
  W_1: (d_ff, d_model)     # up-projection, d_ff usually 4 × d_model
  W_2: (d_model, d_ff)     # down-projection

With d_ff = 4·d_model, the FFN holds 2 · 4 · d_model² parameters per layer — roughly two-thirds of a block's parameters in a typical dense model. To make a model "smarter" you mostly grow d_ff, and the FFN balloons. Every token pays for the entire FFN on every layer. That coupling — bigger model means proportionally bigger per-token compute — is the wall MoE was invented to break.

Dense scaling ties capacity to cost: double the parameters, double the FLOPs per token. MoE breaks the tie. You add many FFNs but route each token to only a few — capacity grows, per-token compute barely moves.

The MoE idea: many experts, few active

Replace the single FFN with N parallel FFNs, called experts. Add a small router (a gating network) that, for each token, picks the top-k experts to actually run. The token is processed only by those k experts; the other N − k sit idle for that token.

Dense FFN:  every token → 1 big FFN              (all params active)

MoE layer:  every token → router picks k of N experts
            token processed by k small FFNs only   (k/N params active)

The router outputs a score per expert; the top-k experts run, and their outputs are combined weighted by the router's gate values:

scores      = softmax(W_router · x)        # (N,) one score per expert
top_k       = indices of k largest scores
y = Σ_{i in top_k}  gate_i · Expert_i(x)   # weighted sum of chosen experts

Concrete numbers from DeepSeek-V2: 160 routed experts, only 6 active per token (plus shared experts, covered in 5.6). The model has the parameter count of all 160 experts but the per-token compute of about 6. That's the whole trick.

Dense vs sparse: the FLOP accounting

Make the tradeoff concrete. Suppose each expert is the same size as the original dense FFN.

Dense model:
  params (FFN)   = P
  active/token   = P
  FLOPs/token    ∝ P

MoE, N=64 experts, k=2 active:
  params (FFN)   = 64 · P        (64× the capacity)
  active/token   = 2 · P         (only 2 experts run)
  FLOPs/token    ∝ 2 · P         (32× cheaper than dense-equivalent capacity)

A 64× larger parameter store, but each token does only 2× the original FFN work. The model holds far more knowledge — different experts specialise in different patterns — while inference and training compute per token stay near a small dense model. This is why "total parameters" and "active parameters" are reported separately for MoE models: DeepSeek-V3 is 671B total, ~37B active per token.

Why specialisation helps, intuitively

A dense FFN must be a jack of all trades — the same weights handle code, prose, math, and dialogue. An MoE layer lets different experts specialise: some lean toward syntax, some toward factual recall, some toward numerical patterns. The router learns to send each token to the experts most useful for it. Capacity isn't just bigger; it's partitioned, so the model can hold more specialised knowledge without every token paying for all of it. Article 5.3 visualises what experts actually specialise in — the picture is messier and more interesting than "expert 7 = Python."

The two costs MoE introduces

Sparse activation isn't free lunch — it trades compute for two new problems that the rest of Phase 5 is largely about solving.

Cost 1: memory

All N experts must be stored in memory even though only k run. DeepSeek-V3's 671B parameters all sit in GPU memory; sparsity cuts compute, not storage. This forces expert parallelism — sharding experts across GPUs — covered in Phase 8.

Cost 2: load balancing

The router is learned, and learned routers collapse. Early in training a few experts get slightly favoured, so they get more tokens, so they train faster, so they get favoured more — a rich-get-richer spiral. Left alone, the model uses a handful of experts and ignores the rest, wasting the capacity you paid for. Worse, in distributed training, uneven token counts per expert mean some GPUs sit idle waiting for overloaded ones. Load balancing — keeping token assignment roughly even across experts — is the central engineering challenge of MoE, and it drives articles 5.4 (auxiliary loss), 5.5 (capacity factor), and 5.6 (DeepSeek's auxiliary-loss-free scheme).

Routing collapse is the MoE failure mode. The router has no built-in reason to spread tokens evenly, and the feedback loop actively concentrates them. Every MoE balancing technique exists to fight this one spiral.

A short history

• 1991 — Jacobs & Hinton introduce mixtures of experts for neural nets. The idea predates Transformers by decades.
• 2017 — Shazeer et al., Outrageously Large Neural Networks, put sparsely-gated MoE into LSTMs at scale (137B params), introducing top-k routing and the auxiliary load-balancing loss.
• 2020–2021 — GShard and Switch Transformer bring MoE to Transformers; Switch simplifies to top-1 routing and shows MoE scales cleanly to trillions of parameters.
• 2024 — DeepSeekMoE introduces fine-grained expert segmentation and shared experts; DeepSeek-V3 adds auxiliary-loss-free balancing. These are Phase 5's destination (5.6).

What's coming in Phase 5

• 5.2 Routing — how the gating network actually works: top-k, softmax-before-or-after, noisy gating, token-choice vs expert-choice.
• 5.3 Visualizing Experts — what specialisation really looks like; why it's distributed, not clean.
• 5.4 Auxiliary Loss — the classic load-balancing loss and the gradient interference it causes.
• 5.5 Capacity Factor — expert capacity, token dropping, and the throughput tradeoff.
• 5.6 DeepSeekMoE — fine-grained experts, shared experts, auxiliary-loss-free balancing.
• 5.7 Code — a complete MoE layer in PyTorch, router to combine.

By the end you'll understand not just what an MoE layer computes, but why DeepSeek's specific variant — 256 fine-grained experts, shared experts, bias-based balancing — outperforms the textbook design. Phase 6 then turns to the other DeepSeek training innovation, Multi-Token Prediction.

References

• Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538
• Fedus et al. (2021), Switch Transformers. arXiv:2101.03961
• Lepikhin et al. (2020), GShard. arXiv:2006.16668
• Dai et al. (2024), DeepSeekMoE. arXiv:2401.06066