The intuitive story of MoE is "expert 7 handles Python, expert 22 handles French." It's a useful first picture and almost entirely wrong. This article looks at what experts actually specialise in, why the clean story fails, and what the real routing patterns imply for how you design an MoE — which is exactly the reasoning behind DeepSeek's fine-grained + shared design (5.6).
The myth: clean semantic experts
You'd hope a trained router sorts tokens by topic — code to one expert, biology to another, so you could point at expert 12 and say "that's the legal-text expert." Researchers looked. With rare exceptions, it isn't there. Specialisation is real but it doesn't line up with human-legible categories like topic or language.
What experts actually specialise in
When you tally which tokens hit which experts across a large corpus, the patterns that emerge are mostly low-level and syntactic, not semantic:
- Token-level / lexical — experts fire on punctuation, whitespace, numbers, or specific frequent tokens. The Switch Transformer authors found experts specialising in things like "the token after a quotation mark."
- Positional and syntactic — experts keyed to part-of-speech-like roles or position in a clause, rather than meaning.
- Distributed semantics — to the extent topic matters, it's spread across many experts in combination, not localised to one. No single "physics expert."
Specialisation is genuine — routing is far from random, and ablating the wrong expert for a token hurts — but it's specialisation in the model's internal feature space, which doesn't map onto the categories humans name.
Experts specialise in features, not topics. The router learned a partition of the model's latent space that's useful for prediction, and that partition mostly tracks low-level token/syntax structure, not "Python vs French." The clean-category picture is a human projection onto a machine's bookkeeping.
Why the clean story can't hold
Two structural reasons specialisation stays distributed and low-level:
1. Per-layer, per-token routing
Routing happens independently at every MoE layer and for every token. A sentence about astrophysics isn't sent to "the astrophysics expert" — each of its tokens is routed separately at each layer, hitting dozens of different experts on the way up. "Which expert handles this sentence" isn't even a well-posed question; there are dozens of decisions per token across depth.
2. Load balancing actively prevents clean clusters
This is the deep reason, and it ties straight to 5.4. Training pressures the router toward even token distribution across experts. A clean semantic partition would be wildly unbalanced — far more tokens are prose than are, say, chemical formulae — so a topic-sorted router would overload some experts and starve others. The balancing objective explicitly pushes against that. The model is rewarded for spreading load evenly, which fragments any would-be clean semantic clusters into distributed, balanced ones. Interpretability and balance are in direct tension.
Fig 1 — Balancing forces even per-expert load (right). A clean topic-sorted router (left) would be wildly unbalanced and is selected against during training.
How researchers actually probe experts
Methods used to study expert specialisation, useful to know if you debug your own MoE:
- Routing heatmaps — for a fixed input set, a matrix of token × expert selection frequency. Reveals whether load is balanced and whether any expert is dead (never selected).
- Top-token analysis — for each expert, the tokens that route to it most often. This is how the "punctuation expert" findings surfaced.
- Expert ablation — zero out an expert and measure loss change per token type. Identifies what an expert contributes without relying on a clean label.
- Co-activation graphs — which experts tend to be selected together across layers, exposing the distributed nature of specialisation.
What this means for design
The empirical picture directly motivates DeepSeek's choices in 5.6:
- Shared experts. Since much of what any token needs is common, low-level processing (the kind a balanced router fragments), DeepSeek dedicates a few always-on shared experts to that common knowledge. Routed experts are then freed to capture the finer, rarer distinctions — pushing specialisation to be more meaningful.
- Fine-grained experts. If specialisation is distributed across many experts anyway, make experts small and numerous (256 vs the classic 8–64). More, smaller experts give the router a finer palette to combine, improving the effective specialisation per token without raising active compute.
In other words, DeepSeek didn't fight the messy reality — it designed around it. Shared experts absorb the common load that would otherwise smear across everything; fine-grained routed experts exploit the fact that real specialisation is distributed and combinatorial.
Don't over-interpret
A practical caution. It's tempting to read deep meaning into routing patterns, but most apparent "specialisation" is the model's internal bookkeeping, not human-meaningful structure. A dead expert (zero load) is a real bug worth fixing; "expert 12 seems to like commas" is mostly trivia. Use expert visualisation to diagnose balance and dead experts — not to tell stories about what the model "knows."
Next, 5.4 makes the balancing pressure concrete: the auxiliary load-balancing loss, exactly how it pushes load even, and the gradient interference that pushed DeepSeek to abandon it for a bias-based scheme.
References
- Fedus et al. (2021), Switch Transformers — expert specialisation analysis, token-level findings. arXiv:2101.03961
- Jiang et al. (2024), Mixtral of Experts — routing analysis showing syntactic, not topical, specialisation. arXiv:2401.04088
- Dai et al. (2024), DeepSeekMoE — fine-grained + shared experts motivated by specialisation structure. arXiv:2401.06066