Pre-training Under Infinite Compute — Optimizing for Data, Not FLOPs.

This one is pitched deeper than the rest of the series — written for readers comfortable with scaling laws, regularization, and the language of asymptotics. If you do LLM research, this is the paper in the batch most likely to change how you set up an experiment.

The premise is a clean inversion of the last decade. The Chinchilla era asked: given a fixed compute budget, how should I split it between model size and data? Compute was the scarce resource. But compute is growing far faster than the supply of high-quality web text. So Konwoo Kim, Suhas Kotha, Percy Liang, and Tatsunori Hashimoto (arXiv:2509.14786) flip the constraint: data is fixed, compute is effectively infinite. Now what?

The regime shift: stop optimizing loss per FLOP and start optimizing loss per token of data. When compute is free, the only scarce thing left is information, and the entire recipe changes.

The first observation: naive scaling overfits

The obvious move under infinite compute is "just train a bigger model longer on the data you have." The paper shows this eventually overfits. Past a point, more parameters and more epochs on a fixed corpus drive training loss down while validation loss turns back up. The standard data-constrained playbook hits a wall — it does not monotonically benefit from extra compute.

So "infinite compute" is not a free lunch. The question becomes: what knobs convert surplus compute into lower asymptotic loss on fixed data, rather than into overfitting?

Background: Chinchilla and the data wall

To feel the inversion, recall the Chinchilla result (2022). Given a fixed compute budget $C \approx 6ND$ (for $N$ parameters and $D$ tokens), it asked how to split $C$ between $N$ and $D$ to minimize loss, and found you should scale them together — roughly 20 tokens per parameter. The whole framing assumes compute is the binding constraint and data is abundant enough to feed whatever model size you pick.

That assumption is breaking. Compute (GPUs, FLOPs) keeps growing exponentially; high-quality human text does not. Estimates put the stock of useful public text in the low tens of trillions of tokens — and frontier runs are already there. So the realistic future constraint flips: you have more compute than you can spend without re-reading the same tokens many times. Chinchilla-optimal advice ("just scale $N$ and $D$ together") becomes inapplicable because you can't scale $D$. The question is no longer "how to spend compute" but "how to extract maximum signal from a fixed $D$ when compute is nearly free."

Lever 1 — regularization, turned way up

The headline practical finding. To stop the overfitting, the authors retune regularization — and the optimum is dramatic: weight decay roughly 30× larger than standard practice.

The intuition: standard weight-decay values were tuned in the compute-bound regime, where you take relatively few passes over a lot of data. In the data-bound regime you hammer the same tokens many times, so you need a much stronger pull toward simple solutions to avoid memorizing. Most people would never sweep weight decay that high because in the old regime it would underfit; here it's exactly right.

optional read — why heavier decay, formally

Weight decay $\lambda$ adds $\tfrac{\lambda}{2}\|\theta\|^2$ to the loss, biasing toward small-norm solutions. The effective regularization strength a parameter "feels" scales with the number of gradient updates it participates in. In the data-rich regime, the per-token update count is low; in the data-poor / many-epoch regime it is high, so the same generalization target requires a much larger nominal $\lambda$. Empirically the authors find the loss-minimizing $\lambda$ is ~$30\times$ the conventional value, and — importantly — that with this retuning the loss as a function of parameter count $N$ follows a clean power law down to a finite asymptote $E$:

$$L(N) \;=\; E + \frac{A}{N^{\alpha}}$$

rather than U-turning into overfitting. The asymptote $E$ — not the loss at any one budget — becomes the quantity to optimize.

Lever 2 — fit the asymptote, don't pick a budget

This is the methodological core, and it's the part PhD readers should steal. Instead of evaluating models at one chosen size/compute budget, the authors fit power-law scaling laws that predict how loss falls as parameters grow, and then read off the asymptotic loss — the floor the method converges to as compute → ∞.

Why this matters: two recipes can look identical at a small budget but have very different floors. Comparing methods by their fitted asymptote $E$ is the right comparison under infinite compute, because the budget you happen to run at is arbitrary. It turns "which is better?" into "which has the lower limit?" — a much more stable and meaningful question.

Lever 3 — ensembles beat single models at the limit

A single regularized model has a floor. The authors show that ensembles of independently trained models reach a lower loss asymptote than any single regularized model can. With surplus compute, training $K$ models and combining them converts compute into a strictly better floor — a different and better use of FLOPs than making one model ever bigger.

Combined with parameter scaling and regularization, this gives two complementary axes of "spend compute to lower the data-bound floor": grow each model (parameter scaling) and grow the number of models (ensemble scaling). Both have their own power-law behavior.

Putting the levers together: the data-efficiency result

The payoff is stated as data efficiency, which is the right currency in this regime. Combining epoching + regularization + parameter scaling + ensemble scaling, the authors match a baseline's loss using dramatically less data:

• Match the baseline at 200M tokens using 5.17× less data.
• 9% improvement on pre-training evaluations.
• 17.5× data-efficiency improvement over continued pre-training on math mid-training data — i.e. when you have a small, precious domain corpus, these techniques stretch it enormously.

Read those numbers as: the same fixed pile of tokens, squeezed for far more. In a world where high-quality text is the bottleneck, multiplying its effective value is worth more than another rack of GPUs.

Lever 4 — distill the ensemble back down

The obvious objection: ensembles are expensive to serve. You don't want $K$ models in production. So the authors distill the ensemble into a single student 8× smaller, retaining 83% of the ensemble's benefit.

This closes the loop. Use infinite compute at training time — heavy regularization, parameter scaling, ensembling — to push the data-bound loss floor as low as possible, then distill that capability into a compact model that's cheap to deploy. The expensive part stays offline; the served artifact is small.

Why ensembles win at the limit — the bias–variance view

The ensemble result isn't a free-floating trick; it falls out of bias–variance. A single model's error decomposes into bias (systematic error from the model class) plus variance (how much the fit wobbles with training randomness) plus irreducible noise. Heavy regularization fights variance by constraining the model — but past a point it adds bias, and that's the floor a single regularized model can't get under.

Averaging $K$ independently trained models attacks variance from a different direction: their individual errors are partly uncorrelated, so the average cancels the wobble without adding the bias that more regularization would. That's why an ensemble's asymptote sits below any single regularized model's — it's buying variance reduction that regularization can only get by paying in bias. Under infinite compute, training many models is cheap, so this is a natural place to spend the surplus.

The full recipe, stacked

The levers compose, and the order tells a story. Read it as a checklist for data-bound pre-training:

1. Epoch the data — accept multiple passes over the fixed corpus (the only way to spend more compute on fixed $D$).
2. Crank regularization — retune weight decay (~30× standard) so epoching doesn't overfit. This is what makes the loss-vs-size curve monotone instead of U-turning.
3. Scale parameters — grow $N$ along the now-clean power law toward its asymptote $E$.
4. Scale ensembles — train $K$ models and average, pushing the asymptote below the single-model floor.
5. Distill — compress the ensemble into one student ~8× smaller, keeping ~83% of the gain, so serving stays cheap.

Each step exists to unblock the next: you can't safely epoch without heavy regularization, the power law only appears once overfitting is tamed, ensembling only helps once each member is well-regularized, and distillation only matters because the ensemble is otherwise unservable.

The data-efficiency numbers, in context

The payoffs — match baseline at 200M tokens with 5.17× less data, a 9% eval improvement, and 17.5× data efficiency over continued pre-training on math — all say the same thing in different units: the same tokens, worth much more. The math number is the most pointed. Continued pre-training on a small, precious domain corpus (where you genuinely cannot get more tokens) is exactly the data-bound regime in miniature, and a 17.5× stretch there is the difference between a domain adaptation that works and one that overfits and fails.

Why this is an important paper for researchers

Three reasons it's worth more than its abstract:

1. It names the regime change. The field's mental model is still Chinchilla-shaped (compute-bound). As frontier labs run into the data wall, the data-bound / compute-rich regime is where the action moves, and this paper is an early, careful map of it.
2. It promotes a better evaluation methodology. Comparing fitted asymptotes instead of point estimates at an arbitrary budget is a more honest way to compare methods, and it generalizes well beyond this paper. That alone is a useful import into your own experimental design.
3. The 30× weight-decay finding is immediately actionable. If you're doing data-constrained pre-training or domain continued-pretraining, your regularization is almost certainly tuned for the wrong regime. This is a concrete knob to sweep tomorrow.

The honest caveats

• "Infinite compute" is a modeling idealization. Nobody has infinite compute; the results are about the limit and the scaling-law extrapolations toward it. How well the fitted asymptotes hold at genuinely frontier scale is an empirical bet.
• Scale of experiments. The studies are run at academic scales (hundreds of millions of tokens / modest models). The power-law framing is explicitly designed to extrapolate, but extrapolation is extrapolation — the conclusions are strongest as directional guidance.
• Ensembling + distillation has real cost. The training-time bill for ensembles is large; the method trades training compute for data efficiency, which is exactly the right trade if your true constraint is data and you genuinely have surplus compute. If you don't, the calculus is different.

The throughline of the whole paper is a single reframing: when compute stops being the bottleneck, every default we inherited from the compute-bound era — model-vs-data splits, weight-decay values, single-model training, point-budget evaluation — is potentially mis-tuned. This paper re-derives the defaults for the regime we're actually heading into.

arXiv:2509.14786 — Pre-training under infinite compute, Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto. Presented at the YC Paper Club. This closes the YC Paper Club series.