Capacity Factor & Token Dropping.

The aux loss (5.4) makes load roughly even. But hardware needs more than roughly — it needs fixed-size buffers known before the batch runs. That requirement is where capacity factor and token dropping come from. This article is the systems half of MoE balancing: why experts have a hard buffer size, what happens to the tokens that don't fit, and how DeepSeek arranges things so almost nothing gets dropped.

Why experts need a fixed buffer

On a GPU, the expert computation is a batched matmul. Batched matmuls want a static shape: expert tensors are [N_experts, capacity, d_model]. That capacity dimension has to be a fixed integer chosen before you see how routing actually distributes tokens. You cannot grow a tensor mid-kernel because expert 7 turned out popular this step.

So every expert gets the same number of token slots, the expert capacity. Routing is even on average, but never exactly even on any given batch — some experts will be handed more tokens than their slots, others fewer. That mismatch is the entire subject of this article.

The capacity formula

Capacity is the even share scaled by a capacity factor C:

capacity = ceil( C * (T * k) / N )

  T = tokens per batch (per device)
  k = experts per token (top-k)
  N = number of experts
  C = capacity factor (slack), e.g. 1.0, 1.25, 2.0

  T*k / N  = the perfectly-even load per expert
  C        = headroom above even for the experts that run hot

C = 1.0 means "exactly the even share, no slack." C = 1.25 means each expert can absorb 25% more than its fair share before overflowing. The factor buys tolerance for the unevenness the router can't fully remove.

Two failure modes of a fixed buffer

1. Overflow → dropped tokens

If more tokens route to an expert than it has slots, the surplus can't be processed. They are dropped: the expert produces no output for them. The token isn't deleted from the sequence — it skips the expert via the residual connection, passing through that MoE layer unchanged. Its representation simply doesn't get the expert's transformation at that layer.

Dropping is a real information loss and it's biased — the tokens most likely to be dropped are exactly those routed to popular experts, i.e. common patterns. A high drop rate hurts quality and, worse, hurts it unevenly across the data distribution.

2. Underflow → wasted padding

If an expert gets fewer tokens than its capacity, the empty slots are padded with zeros so the tensor shape stays static. That padding is pure wasted compute — you run the matmul on slots that hold nothing. A large C guarantees few drops but pays for it in padding FLOPs on every under-loaded expert, every step.

Capacity factor is a dial between two wastes: too low and you drop tokens (lost quality); too high and you burn compute on padding (lost efficiency). The aux loss and capacity factor are partners — better balance lets you run a smaller C safely.

Training vs inference

The two regimes pull C in opposite directions:

• Training often tolerates a modest drop rate — typical values land around C = 1.0–1.25. Dropping acts a bit like noise/regularisation, and the aux loss is simultaneously driving balance down, so the drop rate falls over the run. Lower C keeps training FLOPs down.
• Inference, especially with batch size 1 or autoregressive decoding, can't afford any dropping on the tokens a user is waiting on. Production serving generally runs drop-free: either a large enough capacity, or a routing scheme that guarantees no overflow.

This train/infer gap is itself a problem — a model trained with dropping behaves slightly differently when served drop-free. Minimising drops in both regimes is preferable, which is the direction DeepSeek pushed.

Expert-choice routing sidesteps dropping entirely

Recall the token-choice vs expert-choice distinction from 5.2. Expert-choice flips the selection: instead of each token picking its top-k experts (which can overload an expert), each expert picks its top-capacity tokens. By construction, every expert ends exactly full — perfect balance, zero drops, zero padding.

The catch is the mirror image: a token might be chosen by many experts or by none. A token no expert selects gets no MoE processing at all — effectively dropped from the other side. Expert-choice trades guaranteed expert balance for no guarantee of per-token coverage, which is awkward for autoregressive decoding where you process one token at a time. Useful tool, not a free lunch.

DeepSeek's approach: balance so well you barely drop

DeepSeek's strategy across V2/V3 is to attack the root cause — make routing balanced enough that a small capacity rarely overflows — rather than papering over imbalance with a big C:

• Device-limited routing. Each token's selected experts are restricted to a small number of devices (V2 caps the target experts to M devices). This bounds the all-to-all communication and tightens the load distribution per device, so device-level capacity is easier to satisfy.
• Strong balancing. The expert/device/communication aux losses in V2 — then the aux-loss-free bias controller in V3 (5.4) — keep realised load close to even, shrinking the overflow that capacity has to absorb.
• No token-dropping in V3. Because the bias-based balancing holds load so tight, DeepSeek-V3 reports training without dropping any tokens — the balance is good enough that capacity isn't the binding constraint. The train/infer gap closes.

The arc of MoE balancing: early models accept dropped tokens and tune a big capacity factor to limit the damage. DeepSeek instead drives balance hard enough — device-limited routing plus bias control — that it can train drop-free. Capacity stops being a quality dial and becomes just a safety margin.

Practical guidance

• Monitor the drop rate as a first-class training metric. A rising drop rate is an early warning of balance problems, often before loss curves show it.
• Don't fix imbalance with capacity. A huge C hides a balance bug behind padding FLOPs. Fix the balance (aux loss / bias / routing) and let C stay small.
• Match train and inference. If you serve drop-free, aim to train near drop-free too, so behaviour transfers.
• Account the padding. Effective MoE FLOPs include padded slots. A model that looks cheap on paper can be much more expensive at C = 2.0.

That completes the mechanics of MoE — routing, visualisation, balancing loss, and capacity. Next, 5.6 assembles all of it into DeepSeekMoE: fine-grained experts, shared experts, and the specific architecture that made these ideas work at frontier scale.

References

• Lepikhin et al. (2020), GShard — capacity factor and token-dropping mechanics in expert parallelism. arXiv:2006.16668
• Fedus et al. (2021), Switch Transformers — capacity factor trade-offs and drop-rate analysis. arXiv:2101.03961
• Zhou et al. (2022), Mixture-of-Experts with Expert Choice Routing — drop-free expert-choice selection. arXiv:2202.09368
• DeepSeek-AI (2024), DeepSeek-V2 — device-limited routing and balancing. arXiv:2405.04434
• DeepSeek-AI (2024), DeepSeek-V3 — drop-free training via bias-based balancing. arXiv:2412.19437