Frontier Digest #7 — Models Compete. Runtimes Decide.

The model leaderboard was not the most useful scoreboard this week. OpenAI reportedly slowed parts of its unreleased Astra program after cyber evaluations raised the possibility of critical capability, while Washington discussed a private pre-release review framework. In research, the same systems question appeared from every direction: where does the model end and the runtime begin? A 41-mode taxonomy assigns failures to the component that can repair them; Harness-R1 learns to patch executable scaffolding; DataSpace shows a 15.36-point accuracy swing from harness choice with the backbone fixed; Prompt-Induced Waste finds 5–30× differences in cost per success across harnesses. Memory, reflection, skill libraries, long-horizon planning, and even tokenization all land on the same conclusion: the model supplies capability, but the runtime determines how much useful capability survives contact with the task.

The five signals

1. Astra became a release-process story before it became a product. Axios reported that OpenAI slowed parts of the unreleased model's development after internal testing could not rule out critical cyber capability. OpenAI had not published a model card or launch artifact during the window. reported
2. Frontier evaluation is moving inside government review. A second Axios report described a voluntary, non-public U.S. framework for evaluating certain closed frontier models before release, including a reported 30-day review period. reported
3. Agent failures finally got an address. “Model or Harness?” defines 41 interaction-level failure modes and names the side responsible for repair, replacing the useless diagnosis that “the agent failed.” paper
4. Harness engineering became trainable. Harness-R1 uses failure trajectories to teach a separate 9B model to edit the executable runtime, improving a frozen target agent by 9.3 percentage points across three environments. paper
5. Efficiency moved above the model layer. Prompt wording can multiply reasoning spend by 2.4–7.4×, harness choice can shift cost per successful task by 5–30×, and TokTier shows tokenization consuming up to 64% of time to first token in high-cache-hit agent traffic. papers

Lead story — capability is a property of the system

For years, the industry used a simple mental model: intelligence lives in the checkpoint, and everything around it is plumbing. Pick the strongest model, connect a few tools, add retrieval, and the product inherits the model's capability.

That abstraction is now breaking. An agent does not receive a task and emit one answer. It repeatedly crosses boundaries: the harness constructs context, the model proposes an action, a tool adapter translates it, the environment returns an observation, memory decides what persists, and a verifier decides whether the run is complete. Each boundary can add capability, suppress it, waste it, or direct it at the wrong target. The observable unit is not the model. It is the model–runtime–environment system.

This is why this week's Astra reporting matters even without a launch. A capable cyber model is not assessed only by asking what it knows. Evaluators must decide which harness elicits the strongest credible behavior, what tools and budget it receives, which safeguards remain active, and how the environment is isolated. Change those conditions and the measured capability changes. A pre-release decision made from a weak harness may understate risk; one made from an unrealistic super-harness may overstate normal product exposure. Either way, the harness becomes part of the policy evidence.

Release governance: Astra and the private-review problem

The research map

#	Paper	Theme	The useful result
1	Model or Harness?	diagnosis	Maps 41 failures to the interaction edge and component that can repair them; automated judges reach κ=0.76 against human labels.
2	Harness-R1	runtime learning	A trained 9B harness engineer patches executable runtimes from failure trajectories and lifts frozen-agent success by 9.3 points.
3	DataSpace	data agents	Across 410 heterogeneous analytics tasks, harness choice creates a 15.36-point spread with the same backbone.
4	Prompt-Induced Waste	efficiency	Prompt wording multiplies reasoning without improving success; harness choice moves cost per success by 5–30×.
5	Zero-Mem	memory	Keeps raw traces, indexes entity and temporal views, and removes LLM calls from memory operations while cutting their time cost 57.6%.
6	Sample More, Reflect Less	reasoning	At matched token cost, no tested reflection method reliably beats repeated sampling; all 18 self-inspection comparisons are negative.
7	Rehearse	autoresearch	Focused memory of earlier attempts restores late-stage proposal judgment from a 56.9% confidence cliff to 83.5% selective accuracy.
8	ContinualSkillBench	skills	Explicit skill libraries match plain in-context learning on average; weaker models collect more fragmented, task-specific skills.
9	MerchantBench	long horizon	In a 365-day business simulation, the best agent configuration reaches only 27.3% of human mean final net assets.
10	TokTier	serving	Exact stateful tokenization cuts median time to first token 16–34% in agent workloads without changing emitted token IDs.

1. Stop saying “the agent failed”

Model or Harness? — make failure ownership actionable

An agent misses a task. Was the model unable to reason, or did the harness omit the needed observation? Did the tool adapter corrupt an argument, did memory retrieve stale evidence, did the environment behave differently from the benchmark, or did the grader reject a valid answer? Outcome labels cannot tell you.

“Model or Harness?” represents an agent as interacting components—model, harness, user, tools, memory, and environment—and assigns each of 41 failure modes to an edge plus a fault side. That construction turns taxonomy into an engineering queue. Model-side errors become post-training candidates; harness-side errors become context or integration fixes; grader and environment errors become benchmark repairs. Across four frontier models, the strongest automated judge reached Cohen's κ of 0.76 against human category labels, enough to suggest continuous production labeling rather than one-off postmortems.

Harness-R1 — let the runtime learn from the trace

Harness-R1 takes the next step: instead of only labeling a failure, it trains a separate model to edit the executable harness. A 9B “harness engineer” receives batches of failed trajectories and proposes validated runtime patches. The target agent stays frozen; fresh reruns provide the reward, so improvement is attributable to the runtime edit rather than a moving model.

Across WebShop, ALFWorld, and DBBench, vanilla Qwen3.5-9B rises from 44.3% to 53.6%. After the target itself is fine-tuned, a target-specific engineer lifts the average again from 59.2% to 64.2%. This does not prove autonomous harness editing is ready for unsupervised production—an executable patch can introduce security and distribution-shift failures that the benchmark never sees. It does prove that failure traces contain learnable infrastructure improvements, not only training data for the base model.

2. Cost is designed before inference begins

Prompt-Induced Waste — language is a resource policy

The common advice to “think deeply,” “consider several approaches,” or seek “maximum certainty” sounds harmless. In 4,643 valid runs across six reasoning models, 24 deterministic coding tasks, and two agent harnesses, it was expensive without being useful. Asking for multiple approaches increased reasoning tokens by 2.4–7.4×. Generic deep-thinking cues raised deliberation 1.6–2.2×. Neither effect created a success gradient.

The bigger result sits one layer up: identical model–task–prompt triples cost 5–30× more per successful task in one harness than another, driven largely by static prefixes and additional turns. A bounded-efficiency prompt—scope, acceptance criteria, and a stop condition—was neutral or better on held-out models and could halve reasoning. This is a direct operations lesson: token budgets are not only provider settings. They are encoded in prompt phrasing and loop policy.

Sample More, Reflect Less — reflection needs a fair baseline

Reflection methods spend extra tokens criticizing, revising, debating, and checking. To know whether the method works, the comparison must give a simpler baseline the same budget. Across 36 matched comparisons using 1.5B, 3B, and 7B open models on two math benchmarks, no method reliably beat repeated sampling at equal token cost. Ten were significantly worse; all involved a model inspecting its own output. Every one of the 18 self-inspection comparisons was negative.

The practical conclusion is not “never reflect.” It is “do not call additional text evidence of additional intelligence.” Compare against repeated independent attempts under the same total tokens, and prefer an external verifier whenever correctness can be checked mechanically.

TokTier — the invisible CPU tax in agent loops

Serving stacks cache prompt KV state but often tokenize the full request again on every call. Coding agents make the inefficiency pathological: a long transcript returns with a small tool-result append, over and over. Across 153,951 real calls at a 94.1% prompt-cache hit rate, tokenization grew to as much as 64% of time to first token.

TokTier re-tokenizes a small window near the append boundary and splices only when a stability check proves the token IDs match full reference tokenization; otherwise it widens the window or falls back. Across 17 tokenizer families and 15 billion split checks, the authors report zero divergence. Integrated with vLLM, median time to first token falls 16–34%. The larger lesson: after model inference is optimized, ordinary preprocessing becomes the bottleneck. Agent latency is a full-stack problem.

3. Memory should preserve evidence, not manufacture it

Zero-Mem — generation is not required for organization

Many agent-memory systems ask an LLM to summarize each interaction, write a memory, and rerank it later. Every operation adds tokens and latency, while summaries quietly discard details. Zero-Mem keeps original traces as the source of record and constructs two non-generative views: an entity–context graph for cross-session relationships and a temporal hierarchy for local sequence and state.

At query time, it retrieves from both, follows their structure, and removes conflicting evidence before a single final reader call. With the reader and context budget held constant, memory-operation time falls 57.6% against the fastest compared baseline while accuracy remains competitive. The insight is architectural: use deterministic structure to narrow evidence, then spend generation only where interpretation is necessary.

Rehearse — memory must get more selective as progress accumulates

Autoresearch loops improve code through repeated propose–train–measure cycles. Early changes are easy; late changes interact with everything already accepted. Public logs show useful proposals falling from 70% in the first two iterations to 43% from iteration six onward. An LLM judge's selective accuracy drops from 82.8% early to 56.9% late while its willingness to decide stays high—the “confidence cliff.”

Rehearse proposes several ideas, predicts before executing, and compares against a focused memory of similar previous attempts and outcomes. Late selective accuracy recovers to 83.5%, and 4,000 budgeted training runs across three loops finish at better endpoints under the same run budget. Raw history is not enough; useful memory is history selected for the current decision boundary.

ContinualSkillBench — a growing skill folder can hide failed abstraction

ContinualSkillBench spans five domains with 100 connected tasks each, ordered to offer repeated opportunities for reuse. Sequential experience usually helps, but explicit skill libraries perform comparably to plain in-context learning on average. Skills help selectively on procedures and precision-sensitive outputs; they are not a universal compounding mechanism.

The sharpest diagnostic is collection shape. Weaker models accumulate larger, more fragmented sets of task-specific instructions. That looks like learning from the outside—more files, more names, more text—but often represents failure to extract a reusable procedure. Measure reuse and transfer, not skill count.

4. Better benchmarks make the runtime visible

DataSpace — analytics is a workspace, not a SQL query

Real analytical work crosses databases, CSV and JSON files, long documents, and video. DataSpace places 410 tasks inside workspaces containing 7,439 artifacts totaling 15.01 GB. The agent receives a question and the workspace, then must return the complete requested table. Evaluation is deterministic: it aligns columns independent of headers, normalizes types and precision, and compares rows with order awareness.

The best system reaches 66.34%, leaving substantial room. More importantly for this edition, switching the harness while keeping the backbone fixed creates a 15.36-point accuracy spread. Joins and multimodal evidence integration hurt all six tested backbones. If a leaderboard names only the model, it hides a variable large enough to reverse conclusions.

MerchantBench — coherence has carrying costs

Most agent benchmarks end before consequences arrive. MerchantBench runs a 365-day e-commerce simulation grounded in 98,843 product records and exposes 26 tools for sourcing, listing, pricing, cash flow, and feedback. Supplier events arrive quickly; order outcomes arrive later. Earlier decisions constrain later options, and inconsistency compounds in final net assets.

Across eight models, two frameworks, and 48 year-long simulations, the best configuration reaches only 27.3% of the human participants' mean final net assets. Long-horizon weakness is not just forgetting a fact. It is failing to maintain policy, revisit old decisions when delayed evidence arrives, and preserve resources for future states.

What matters most

1. Treat model plus harness as the evaluated artifact. Record both versions, plus tools, context policy, memory, verifier, budget, and environment.
2. Localize before you optimize. “Agent failure” is not a diagnosis. Assign the failure to the interaction edge that can repair it.
3. Use cost-matched baselines. Reflection, debate, and extra planning must beat repeated sampling under equal tokens, time, and tool budget.
4. Keep raw evidence. Summaries can be indexes, not sources of record. Let deterministic systems do deterministic memory work.
5. Test delayed consequences. A five-minute task cannot validate a system meant to operate for weeks.

A practical runtime checklist

Caveats & evidence notes

How this digest works

Frontier Digest reads the whole week as a system: releases and reported developments, policy, research, open artifacts, and the operational lessons connecting them. Confirmed marks primary-source publication or an identifiable paper. Reported marks information dependent on journalism or a single non-primary account. A source-backed claim is not automatically an independently reproduced claim. Read every edition →

References & sources

• Axios — OpenAI slows release of Astra model, citing cyber capabilities — reported development slowdown and cyber-evaluation concern.
• Axios — Inside the White House AI framework — reported scope, closed-model focus, and review process.
• arXiv — Model or Harness? — interaction-centric failure taxonomy.
• arXiv — Harness-R1 — learned executable-runtime editing.
• arXiv — DataSpace — heterogeneous workspace benchmark and deterministic evaluator.
• arXiv — Prompt-Induced Waste — preregistered coding-agent cost benchmark.
• arXiv — Zero-Mem — zero-token memory operations over raw traces.
• arXiv — Sample More, Reflect Less — reflection methods under matched token budgets.
• arXiv — Rehearse — confidence cliff and focused outcome memory.
• arXiv — ContinualSkillBench — continual skill learning and transfer.
• arXiv — MerchantBench — 365-day long-term coherence benchmark.
• arXiv — TokTier — exact stateful tokenization for agent serving.