From GPT to recurrent depth: the ten-year story behind one sentence.

At the time, the field's standard practice was one bespoke model per task: a bidirectional LSTM with attention for QA, something different for entailment, something different again for sentiment. GPT-1 is where "just pretrain one general thing and adapt it" starts winning.

And why does predicting the next token work so absurdly well as a general objective? Because to predict what comes next in arbitrary internet text, you have to implicitly model grammar, facts, causal structure, argument, and the personalities of the people who wrote it. The objective is trivial to state, has infinite free supervision (every token is a label), and is bounded above only by how much the world's text actually contains. That combination is what made the next five years possible.

GPT-2: the same thing, ten times bigger, and suddenly it does tasks nobody trained it on Scaled by OpenAI

Language Models are Unsupervised Multitask Learners (Radford et al., 2019) is architecturally almost identical to GPT-1. 1.5B parameters, 40GB of curated web text, and a title that is a literal description of the finding.

The model, trained on zero task-specific supervision, started doing summarization, translation, reading comprehension and QA in a pure zero-shot setting. Nobody put those tasks in. They were latent in "predict the next token across a big enough slice of the internet," and scale surfaced them.

This is the moment the field's center of gravity visibly shifts. Before GPT-2: capability comes from architecture. After: capability might just come from size.

Sparse Transformer: the real architecture invention nobody talks about Invented by OpenAI

Here is OpenAI's cleanest, most under-discussed architecture contribution — Generating Long Sequences with Sparse Transformers (Child, Gray, Radford, Sutskever, 2019).

Self-attention compares every position to every other position. That's \(O(N^2)\) in both time and memory. Fine for 512 tokens. Catastrophic for the tens-of-thousands-long sequences you get from raw audio or high-resolution images.

Their fix: don't attend to everything. Factorize attention into a couple of cheap sparse patterns — a strided pattern that captures local structure, plus a fixed pattern where positions attend to a small set of summary columns that carry global context. Stack them across layers and information still reaches everywhere, but the cost drops to roughly

They also shipped the boring-but-essential parts: recomputing attention during the backward pass to save memory, and initialization changes to train much deeper stacks. Same architecture, applied to text, images, and raw audio bytes — state of the art on Enwik8, CIFAR-10, and ImageNet-64 simultaneously, plus a demonstration that million-token attention was possible in principle.

The paper that turned research strategy into arithmetic Invented by OpenAI prescription later corrected

Scaling Laws for Neural Language Models (Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei, 2020) is one of the most strategically consequential papers ever published in ML, and almost none of its specific numbers survived intact. Both of those things are true.

The finding: cross-entropy loss follows clean power laws in parameters \(N\), data \(D\), and compute \(C\), holding over seven orders of magnitude.

And — the part that changed behavior — architectural details like width vs. depth barely mattered once you fixed total parameter count. Their fitted exponents said that for a fixed compute budget you should spend it disproportionately on parameters: \(N_{\text{opt}} \propto C^{0.73}\), \(D_{\text{opt}} \propto C^{0.27}\).

Before: invent a smarter architecture.
After: hold the architecture still and ride a curve you can extrapolate.

That's the intellectual justification for spending GPT-3 money. It's also, in its prescriptive form, wrong.

Two years later Chinchilla (Hoffmann et al., DeepMind, 2022) showed that Kaplan's training runs were mostly stopped before converging, which biased the fitted optimum toward oversized, undertrained models. Run each size to convergence instead, and the optimum rebalances hard toward data — roughly 20 tokens per parameter. Chinchilla at 70B beat Gopher at 280B and GPT-3 at 175B on equal compute, purely by reallocating.

This is a genuine two-lab correction cycle and it deserves to be told that way: OpenAI established the phenomenon and the methodology; DeepMind fixed its most load-bearing conclusion. Neither had the last word.

GPT-3, and the thing nobody ordered Scaled by OpenAI

The headline for Language Models are Few-Shot Learners (Brown et al., 2020) was 175 billion parameters. The actual story was weirder.

GPT-3 could be shown a few examples of a task in its prompt and then do that task — with no weight update whatsoever. Frozen model. No gradients. Just examples sitting in the context window.

Six years later, why this happens is still not fully settled. Worth being honest about what's established versus what's argued:

Note who's on that list. None of those explanations are OpenAI's. GPT-3's contribution was making a phenomenon so undeniable that everyone else had to go explain it.

The year OpenAI pointed the same hammer at every modality

There's a run of four papers here that look like scattered curiosities and are actually one sustained argument: the architecture isn't about language.

ImageGPT — what if you just… didn't tell it images are 2D? Invented by OpenAI

Generative Pretraining from Pixels (Chen, Radford, Child, Wu, Jun, Luan, Sutskever, 2020) asked a deliberately dumb question: flatten an image into a 1D string of pixels, feed it to the exact GPT recipe with zero knowledge of spatial structure, and see what happens.

text tokens   →  autoregressive Transformer  →  next token
pixel tokens  →  autoregressive Transformer  →  next pixel

What happened: a GPT-2-scale model learned image representations strong enough to be competitive on linear-probe and low-data classification. ImageGPT never became a serious image generator — raw pixel sequences are a terrible inductive bias and the compute cost is brutal. That was never the point. The point was that the recipe wasn't language-shaped.

Jukebox — when your sequence is four million steps long Invented by OpenAI

Text has a merciful property: a paragraph is a few hundred tokens. Jukebox (Dhariwal et al., 2020) had to deal with raw audio, where a three-minute song is millions of samples. No autoregressive model touches that directly.

The solution is a pattern worth internalizing because it recurs everywhere: if the native sequence is too long, learn a compression into a shorter discrete one, and model that instead. Jukebox uses a hierarchical VQ-VAE to squash raw audio into discrete codes at several time resolutions, then runs separate Transformers over the codes — coarsest first, finer levels conditioned on coarser. Out the other end: multi-minute songs, with singing, steerable by artist, genre, and unaligned lyrics.

DALL·E — the sentence and the picture are the same sequence Invented by OpenAI

Zero-Shot Text-to-Image Generation (Ramesh, Pavlov, Goh, Gray, Voss, Radford, Chen, Sutskever, 2021) took the Jukebox move and pointed it at two modalities at once. A discrete VAE turns an image into image tokens. BPE turns a caption into text tokens. Then you concatenate them and train one Transformer over the whole thing as a single undifferentiated stream — the model has no architectural notion of "this part is text, this part is picture" beyond position.

CLIP — the quiet one that ended up in everything Invented by OpenAI

If you had to pick OpenAI's most load-bearing research artifact — the one that ended up inside the most other people's systems — it's probably not GPT. It's CLIP (Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark et al., 2021), trained on 400 million image-caption pairs scraped off the internet.

image  →  Vision Encoder  →  z_i
                                 ↘
                              cosine similarity  →  maximize for matched pairs
                                 ↗
text   →  Text Encoder    →  z_t

Two towers, one shared embedding space, and a symmetric InfoNCE contrastive loss that pulls matched image-text pairs together while pushing apart the \(N-1\) mismatched pairs in the batch:

The payoff is zero-shot classification on label sets CLIP never saw: embed "a photo of a {class}" for each candidate, embed the image, take the nearest. No fine-tuning, no classifier head.

Then it showed up everywhere — multimodal retrieval, the guidance signal in early text-to-image pipelines, the default vision encoder bolted onto vision-language models for years. Contrastive cross-modal learning wasn't new (see SimCLR and a long prior literature), but CLIP is where it worked at a scale and simplicity that made it everyone's default.

Its failure modes are equally well documented and worth stating plainly: web-scraped dataset bias, shortcut learning, robustness gaps under distribution shift, and genuinely poor compositional reasoning — CLIP-family models are notoriously bad at telling "red cube on blue sphere" from "blue cube on red sphere," which suggests something closer to bag-of-concepts matching than understanding.

Diffusion: what OpenAI did, and what it very much did not

Guided diffusion Existing idea, significantly advanced

Say it flatly: OpenAI did not invent diffusion models. The lineage is Sohl-Dickstein et al. (Stanford, 2015) for the original nonequilibrium-thermodynamics formulation, then Ho, Jain & Abbeel (Berkeley/Google, 2020) for DDPM, the paper that made diffusion practically competitive.

What OpenAI contributed, in Diffusion Models Beat GANs on Image Synthesis (Dhariwal & Nichol, 2021), was a serious architecture ablation on the diffusion U-Net plus a new technique: classifier guidance.

The process itself: noise a sample forward until it's indistinguishable from Gaussian, learn to run it backward.

Classifier guidance steers the reverse process toward a class \(y\) by adding a classifier's gradient to the score at each denoising step:

with \(s\) trading diversity for fidelity. And the title was the point: diffusion beat BigGAN-deep on ImageNet FID, matching it with as few as 25 sampling steps. GANs had owned high-fidelity image synthesis for years and diffusion was the elegant-but-losing alternative. This paper is the hinge where that flipped.

Classifier-free guidance — and a correction people get wrong constantly Not OpenAI. Google.

Classifier-Free Diffusion Guidance is Jonathan Ho and Tim Salimans, Google Research, 2022. Not OpenAI. (Salimans had an OpenAI affiliation years earlier; the paper's byline is Google.) It's in this article only because it built directly on — and then completely displaced — OpenAI's classifier guidance, and because it's now inside essentially every text-to-image system on earth, so the attribution is worth getting right.

The idea: skip the separate classifier entirely. Train one model to make both conditional and unconditional predictions (just randomly drop the conditioning during training), then extrapolate between them at sampling time:

No extra classifier to train, no adversarial-gradient weirdness at high guidance, and it works with full text captions rather than just class labels. Stable Diffusion, Imagen, Sora — all of them run this, not classifier guidance.

From "generates text" to "does what you asked"

WebGPT — the first agent, before anyone said "agent" Invented by OpenAI

WebGPT (Nakano, Hilton, Balaji, Wu, Ouyang, Kim, Hesse, Jain, Kosaraju, Saunders et al., 2021) is the moment the relationship changes:

GPT-3 was fine-tuned to drive a text-based browser via a small action set — search, click, scroll, find-in-page, quote — trained by imitation on human browsing demonstrations and then refined with RLHF, where preferences partly judged whether the cited evidence actually supported the answer. That's a real action-observation loop: the model's choices change what it sees next, and its answer has to be grounded in what it found.

InstructGPT — where a 1.3B model beat a 175B one Existing pipeline, industrialized

Training language models to follow instructions with human feedback (Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray et al., 2022) is the paper that took the 2017 preference architecture and made it the industry's default post-training step. Note carefully: this is post-training, not architecture. InstructGPT's network is the same GPT-3 family model.

pretrained model  →  supervised fine-tuning  →  human preference collection
                                                          ↓
        aligned model  ←  RL optimization (PPO)  ←  reward model

The reward model is trained on pairwise comparisons under a Bradley-Terry model — given a preferred response \(y_w\) and a rejected one \(y_l\):

Then PPO optimizes the policy against that learned reward, with a KL penalty back toward the SFT model. That penalty isn't a detail — it's the guardrail against reward hacking, where the policy discovers that some weird off-distribution text scores wonderfully with the reward model while being useless to humans.

The result that made the field pay attention: humans preferred outputs from the 1.3B InstructGPT over the 175B base GPT-3. A hundredfold parameter disadvantage, erased by post-training. That single comparison is why RLHF stopped being optional.

Whisper — 680,000 hours and no architectural cleverness Scaled by OpenAI

Robust Speech Recognition via Large-Scale Weak Supervision (Radford, Kim, Xu, Brockman, McLeavey, Sutskever, 2022) is an ordinary encoder-decoder Transformer. Genuinely unremarkable, architecturally.

What's remarkable is 680,000 hours of messy, multilingual, weakly-supervised web audio, and a multitask tokenization scheme where special decoder tokens select the task — transcribe, translate, identify language, emit timestamps — all as one seq2seq problem. Zero-shot performance often matched models that had been fully fine-tuned on each specific benchmark.

It's the GPT-2 lesson again, in a different modality: broad weak supervision at scale beats narrow strong supervision.

Teaching models to think longer, not just to be bigger

Process supervision — grading the working, not just the answer Invented by OpenAI

Let's Verify Step by Step (Lightman, Kosaraju, Burda, Cobbe et al., 2023) draws a distinction that turned out to matter enormously.

An Outcome Reward Model scores only the final answer: \(R(\text{answer})\). A Process Reward Model scores every intermediate step: \(R(s_1), R(s_2), \dots, R(s_n)\).

Training on MATH with a GPT-4-based model and a lot of human step-level labels, process supervision beat outcome supervision — even at matched human-labeling budget. The reasons go beyond accuracy:

• Credit assignment. One bad step in a sound ten-step derivation shouldn't condemn the whole trajectory. Outcome supervision can't tell the difference. Process supervision localizes the error.
• A free interpretability signal. A PRM tells you which step it thinks is bad.
• Search becomes possible. A PRM scores partial trajectories — which is exactly what you need to guide tree search or prune candidates at inference time, rather than only judging finished answers.

That last bullet is the bridge to o1. The catch, which the paper is candid about: step-level human labels are brutally expensive, which is why later work leans on automated or self-generated process signals.

Consistency models — the same trick, run backwards Invented by OpenAI

Everything else in this act spends more inference compute. Consistency Models (Song, Dhariwal, Chen, Sutskever, 2023) is the one that spends less.

Diffusion:          noise → [step] → [step] → … (50–100+ passes) → image
Consistency model:  noise → [one pass]                          → image

Train a model so that from any point on a noise trajectory it maps directly back to the same clean sample — self-consistency along the path. You can get there by distillation from a trained diffusion teacher, or by consistency training from scratch with no teacher at all. Sampling is then one forward pass, or a handful if you want to trade compute back for quality.

State of the art one-step FID at the time: 3.55 on CIFAR-10, 6.20 on ImageNet 64×64. Later work — continuous-time consistency models (sCM) — kept closing the gap with full multi-step sampling. Think of this as the generative-model twin of the test-time-compute story: same dial, turned the other way.

GPT-4, and the paper that tells you nothing Architecture undisclosed

The GPT-4 Technical Report (2023) is notable in this article for what it withholds. OpenAI states explicitly that it contains no details about architecture, model size, hardware, training compute, dataset construction, or training method — citing competitive and safety considerations.

GPT-4o — killing the three-model pipeline Invented as a system; architecture undisclosed

Before GPT-4o (May 2024), talking to ChatGPT meant a relay race:

Three models, ~2.8s average latency, and — the deeper problem — the LLM in the middle only ever sees flattened text. Tone, emotion, emphasis, the fact that you sighed, whether you were interrupting: all discarded at step one, structurally unrecoverable.

GPT-4o is described by OpenAI as natively handling text, audio, image and video in one model, with audio responses as fast as 232ms (320ms average) — human conversational latency. That's a real capability shift, well documented by demos and latency figures.

But be precise: the product capability is documented; the architectural novelty is not. There's no GPT-4o technical report comparable to CLIP's or Whisper's. Claims about its internal mechanism beyond "natively multimodal" are unconfirmed.

Chain-of-thought — Google's idea, and the foundation of everything after Not OpenAI

Before o1, the credit line that must not blur. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models is Wei, Wang, Schuurmans, Bosma, Ichter, Xia, Chi, Le, Zhou — all Google, 2022.

Prompt the model to write out intermediate steps, and hard multi-step problems get dramatically easier — apparently because each additional token position buys another forward pass of computation, instead of cramming the entire derivation into one.

The follow-ups, also mostly not OpenAI, built the toolkit that o1 would later industrialize: self-consistency (Wang et al., Google, 2022) samples many reasoning paths and majority-votes the answers; Tree of Thoughts and verifier-guided search generalize to explicit search over reasoning trajectories; best-of-N and rejection sampling do the cheap version. All of it is the same bet: spend more compute at inference, get better answers.

o1 — the second scaling axis Existing ideas, scaled into a paradigm

For six years the industry optimized one number:

o1 (2024) reported that performance improved along two independent compute axes — more RL during training, and more time spent thinking before answering:

Per the o1 System Card, the model is trained with large-scale RL to produce a long internal chain of thought before responding — trained, not merely prompted. Reported results: 89th percentile on competitive programming, top-500-in-the-US on AIME, above-PhD-baseline on curated science benchmarks. It also introduced deliberative alignment, where the model reasons explicitly about safety policy inside its chain of thought before answering.

Before o1, inference cost per query was roughly fixed. After o1, the same trained model has a dial: pay more, think longer, get better answers. Which is a product decision, an infrastructure problem, and a pricing model all at once

And meanwhile, the agent loop grew up Scaled by OpenAI

The line from WebGPT runs straight through to today: GPT-3 prompting → WebGPT → function/tool calling → code execution → retrieval → computer use → autonomous agents. Same loop, wider action space:

           LLM
            ↓
        decision
     ┌──────┼──────┬────────┬─────────┐
   search  code  browser   API     computer
     └──────┴──────┴────────┴─────────┘
            ↓
       observation
            ↓
           LLM   (loop)

What's changed since 2021 isn't the loop — WebGPT had that. It's the action space (arbitrary structured function calls, a full code interpreter, raw screen and keyboard control). And the hard problems — planning over long horizons, memory across many steps, tool selection, recovering from a bad action, knowing when to stop — are mostly orthogonal to base-model architecture. There's a serious argument that agent scaffold design now matters as much as model capability for real-world task success.

Recurrent depth: the idea Astra reportedly uses

Now back to that nine-word sentence. To understand what it would mean, you need to understand what recurrent depth actually is — and where it came from, which is not OpenAI.

A normal Transformer's depth is a stack of distinct layers, each with its own weights:

Recurrent depth replaces part of that stack with one block, applied over and over:

The modern template wraps that loop in a non-recurrent entry and exit:

          Input
            ↓
         Prelude          ← ordinary layers, run once
            ↓
    ┌─────────────────┐
    │ Recurrent Block │  ←──┐
    └────────┬────────┘     │   repeat K times
             └──────────────┘   (K can change at inference)
             ↓
           Coda            ← ordinary layers, run once
            ↓
          Output

Where it actually came from

The ancestor is Universal Transformers — Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser, Google, ICLR 2019. Weight sharing across depth, plus Adaptive Computation Time (Graves, 2016) so the model learns per-position how many steps to take before halting. It beat both vanilla Transformers and LSTMs on algorithmic tasks and translation, establishing that depth-recurrence is a real inductive bias and not just a parameter-saving hack.

Then it went quiet for six years, and came back in February 2025 with Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (Geiping, McLeish, Jain, Kirchenbauer, Singh, Bartoldson, Kailkhura, Bhatele et al.) — an academic team, no OpenAI involvement. They trained Huginn-3.5B on 800B tokens on the Frontier supercomputer, weights and training code public.

Their headline claim: unroll the recurrent block deeper at test time, with no retraining, and reasoning performance climbs — in places matching the effective capability of a ~50B model from 3.5B trained parameters. And because the extra computation happens in hidden state rather than in emitted tokens, it needs no chain-of-thought training data, works in small context windows, and — the line that matters most for everything below —

…can capture types of reasoning that are not easily represented in words. Geiping et al., 2025 — on why latent recurrence isn't just cheaper CoT

The parameter-efficiency argument is simple arithmetic: FLOPs scale with \(K\), parameters don't scale with \(K\) at all. You're buying effective depth with sequential compute instead of with weights.

Recurrent depth vs. chain-of-thought, honestly compared

Astra Reported, not confirmed

So: back to where we started. The Information reported — relayed via Techmeme, covered by TechCrunch — that Astra uses recurrent depth: looped Transformer layers recomputing hidden states, described in the reporting as improving cost and performance while obscuring the model's reasoning and making it harder to monitor.

The correct sentence is:

"Astra reportedly uses recurrent depth, according to industry reporting."

Not "OpenAI invented recurrent depth" — demonstrably false, see the Google/academic lineage above. And not "Astra uses recurrent depth" as settled architectural fact, which the sourcing doesn't support. Here is the full state of knowledge as of today:

And resist the temptation to fill that last row in from Huginn-3.5B. That's an independent academic artifact that establishes the technique is plausible and gives us a template. It is not a spec for what OpenAI built.

Why this particular sentence is worth a whole article

Every reasoning model before this one externalizes its thinking as tokens. Imperfectly faithful, yes — CoT faithfulness research has shown repeatedly that the stated reasoning isn't always the operative reasoning. But it's there. It's an artifact. You can read it, log it, run a monitor over it.

Chain-of-thought monitoring — reading a model's reasoning trace to catch it planning around a guardrail or gaming a reward — has become a genuinely useful safety tool precisely because that artifact exists. A model doing a meaningful fraction of its thinking in latent recurrent passes doesn't produce one. Not because it's hiding anything. Because the computation was never in token form to begin with.

What's left is the interpretability toolkit: activation probing (train small classifiers on internal activations to detect properties), the logit lens (project intermediate hidden states through the unembedding to see what they'd predict if you stopped there), and mechanistic interpretability more broadly. These are real tools. They are also considerably less mature than "read the text."

Compute stopped being a constant

Step back from the chronology and one theme runs through all of it. Old neural networks spent roughly fixed compute per input. Everything interesting in the last four years is a mechanism for making that conditional:

Six answers to one question: where should extra computation go, and how much does this input deserve? None of them wins alone, and the current trajectory — o1's RL-plus-search, MoE's parameter conditionality, Astra's reported recurrence — looks like convergence on systems that run several of these at once.

The part your inference bill cares about

If you run models rather than train them, here's the comparison that actually matters.

The sharp question underneath all of that:

Does recurrent depth trade token-level sequential decoding for layer-level sequential computation?

Both are irreducibly sequential — you can't run pass \(k+1\) before \(k\) finishes, exactly as you can't decode token \(t+1\) before \(t\). So recurrence doesn't escape the sequential-latency problem; it relocates it.

Where it plausibly wins: if a recurrent pass buys more "reasoning per unit latency" than generating a token does — no KV write, no sampling step, no risk of an off-distribution token derailing the trace — then trading CoT tokens for passes cuts cache pressure and sequence length at equal quality.

Where it plausibly hurts: every scheduler, batching heuristic and speculative-decoding pipeline in production today assumes the cost structure of token generation. A model whose dominant cost is repeated full-network passes with no token growth doesn't fit those assumptions. Serving it well would need real changes — batching that accounts for variable (possibly adaptive) per-request depth, different autoscaling signals, and a genuinely hard question about latency SLOs when a request's total compute isn't knowable from its prompt. That's an open systems problem, not a solved one.

If you want the concrete grounding for any of this, the PagedAttention deep dive covers the KV-cache mechanics these tradeoffs are denominated in, and the vLLM tuning guide covers what the current schedulers actually do.

The scorecard

The things OpenAI did not invent

This list exists because revisionism happens by omission. Each of these predates or sits outside OpenAI, and each gets its real credit:

• The Transformer — Vaswani et al., Google, 2017. Everything above stands on it.
• Attention — Bahdanau, Cho, Bengio, 2014, predating the Transformer itself.
• Chain-of-thought prompting — Wei et al., Google, 2022.
• Diffusion models — Sohl-Dickstein et al., Stanford, 2015; made practical by Ho, Jain & Abbeel, 2020.
• Mixture-of-Experts — Jacobs, Jordan, Nowlan & Hinton, 1991; revived for deep learning by Shazeer et al., Google, 2017.
• RAG — Lewis et al., Facebook AI Research (Meta), 2020.
• Universal Transformer — Dehghani et al., Google, 2018. Recurrent depth's direct ancestor.
• Recurrent networks — Elman 1990; LSTM (Hochreiter & Schmidhuber, 1997).
• Reinforcement learning — decades older than any lab here.
• Self-play — TD-Gammon (Tesauro, 1992); AlphaGo (DeepMind, 2016) is the closer ancestor of OpenAI Five.
• Contrastive learning — a long lineage; SimCLR and MoCo (2020) are the immediate vision-side context for CLIP.

What OpenAI genuinely did in most of these cases is visible throughout this article: take the idea and demonstrate it at a scale, or in a combination, nobody had shown. CLIP's contrastive objective at 400M pairs. Preference learning applied to a 175B model. An agentic loop pointed at the open web. That's a real contribution. It's a different contribution than invention, and the conflation is what this section exists to prevent.

The whole thing, in one column

2017   PPO ─────────────────────────────── the optimizer that survives noisy rewards
        │
       Human Preference Learning (w/ DeepMind) ── alignment becomes trainable
        ↓
2018   GPT-1 ───────────────────────────── pretrain once, adapt cheaply
        ↓
2019   GPT-2 ───────────────────────────── scale alone unlocks unseen tasks
       Sparse Transformer ──────────────── O(N√N) attention, cross-modal
        ↓
2020   Scaling Laws ────────────────────── strategy becomes arithmetic
       GPT-3 ───────────────────────────── in-context learning, unordered
       ImageGPT · Jukebox ──────────────── the recipe isn't language-shaped
        ↓
2021   CLIP · DALL·E ───────────────────── modality as tokens
       Guided Diffusion ────────────────── GANs lose the crown
       WebGPT ──────────────────────────── the first agent loop
        ↓
2022   InstructGPT / RLHF ──────────────── 1.3B beats 175B
       Whisper ─────────────────────────── weak supervision at 680k hours
       ⚠ Chinchilla (DeepMind) ──────────── corrects OpenAI's scaling prescription
       ⚠ Classifier-free guidance (Google) ─ displaces OpenAI's classifier guidance
        ↓
2023   GPT-4 ───────────────────────────── the disclosures stop
       Process Supervision ─────────────── grade the steps, enable search
       Consistency Models ──────────────── 100 steps → 1
        ↓
2024   GPT-4o ──────────────────────────── kill the 3-model voice pipeline
       o1 ──────────────────────────────── inference becomes a scaling axis
        ↓
2025   Recurrent depth revival (academia) ─ Huginn-3.5B, latent reasoning
        ↓
2026   Astra ───────────────────────────── reportedly recurrent depth.
        ↓                                   the transcript gets thinner.
Future Adaptive computation? ────────────── still an open question

Five eras — and the one that hasn't happened yet

Every piece of evidence in this article points at the same unfinished thing: MoE's conditional parameters, CoT's variable length, o1's learned stopping, Astra's reported recurrence with (per the academic template) possible adaptive halting. Nobody has published a confirmed architecture that runs all of these axes at once. That's the gap.

Eighteen open problems

Written as a menu, mostly for people who need a thesis topic. The recurrent-depth cluster (1–13) is unusually tractable on academic compute, because Huginn's weights and training code are public.

1. Does recurrent depth follow the same predictable power-law scaling as parameter depth?
2. How should a recurrent model learn when to halt — is ACT enough at frontier scale, or must halting be learned via RL against task success?
3. Can recurrent representations diverge, collapse, or cycle after too many iterations, and what reliably prevents it at scale?
4. For a fixed inference budget, what's the optimal split between latent passes and explicit reasoning tokens? Substitutes or complements?
5. Can latent reasoning be reliably interpreted with today's probing tools, or does recurrence need new methods?
6. Can process reward models — built for discrete textual steps — supervise latent computation that has no step structure?
7. How should recurrent computation be scheduled on GPUs? Do we need new batching primitives?
8. Does recurrent depth actually reduce KV-cache pressure in practice, or does the block's working state eat the gain?
9. Does it improve compositional generalization beyond what small controlled benchmarks have shown? (Recent work probes exactly this.)
10. How does depth recurrence interact with MoE routing — compounding or conflicting inductive biases?
11. Can a recurrent block learn a genuine internal algorithm (something like an iterative solver), and can we verify that rather than infer it from benchmark scores?
12. Are recurrent-depth models measurably harder to red-team than CoT models, under controlled conditions rather than architectural argument?
13. Can they learn to allocate compute genuinely proportional to difficulty — and how would you measure that rigorously?
14. Does recurrent depth beat search-based test-time scaling at matched compute, or are they complementary?
15. How do you benchmark reasoning that never appears as text? What would "latent reasoning correctness" even mean operationally?
16. Should frontier safety evaluation formally require latent-computation interpretability?
17. Does inference-time compute have its own predictable scaling law? What's the functional form?
18. What architecture combines MoE, recurrence, memory, tools, search and multimodality coherently — and what happens when several conditional-compute mechanisms interact?

What to do with all this

If you read twenty papers, read these

Then, if you want to go deeper

• In-Context Learning and Induction Heads — Olsson et al., Anthropic, 2022. The best mechanistic account of why ICL works.
• Self-Consistency Improves Chain of Thought Reasoning — Wang et al., Google, 2022.
• Tree of Thoughts — Yao et al., 2023. Search over reasoning trajectories.
• Denoising Diffusion Probabilistic Models — Ho, Jain, Abbeel, 2020. The paper that made diffusion work.
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics — Sohl-Dickstein et al., Stanford, 2015. Where diffusion actually starts.
• Retrieval-Augmented Generation — Lewis et al., FAIR, 2020.
• Outrageously Large Neural Networks (Sparsely-Gated MoE) — Shazeer et al., Google, 2017.
• Trust Region Policy Optimization — Schulman et al., 2015. PPO's harder predecessor.
• FlashAttention — Dao et al., 2022. Exact attention, made IO-aware.
• A Survey on Latent Reasoning — 2025. Situates recurrent depth among its neighbours.
• Adaptive Computation Time for RNNs — Graves, 2016. The halting mechanism Universal Transformer builds on.
• The Information's Astra reporting (via Techmeme) — industry journalism, explicitly not an architecture paper. Labeled as such everywhere in this article.

Related reading here

• 26 LLM Architecture Concepts, Explained — the glossary companion: MLA, MoE, DSA, mHC, and every other mechanism referenced above, with two links each.
• PagedAttention deep dive — the KV-cache mechanics all the serving math above is denominated in.
• vLLM Optimization & Tuning — what today's inference schedulers actually do, and where recurrence would break their assumptions.
• GLM-5.3-Flash architecture teardown — one fully-disclosed frontier model, for contrast with everything undisclosed above.

One last thing

For most of this decade, the question driving AI research was how big. More parameters, more data, more training compute. It was a good question and it produced most of what we have.

The question underneath the last two years is different, and harder: how much computation does this particular problem deserve, and where should that computation happen? In more weights? In more tokens? In more passes over the same weights? In a tool that isn't the model at all?

Astra's nine-word sentence is interesting because it's a specific answer to that question — one that moves computation somewhere we currently have poor instruments for. Maybe it's the right answer. It's cheaper, and the academic results suggest it works.

But it's worth being clear-eyed about the trade being made. For four years, the field got a free gift it never really designed for: models that reasoned in English, where you could just read what they were doing. That was a historical accident of how chain-of-thought happened to work, not a property anyone engineered in.

If the next generation thinks in vectors instead, that gift goes away — and the tools to replace it don't exist yet at anything like the same maturity. That's not a reason to stop. It is a reason to build the instruments before we need them.