The experiment supports a focused claim: on 60 behavior-driven planning requests across Codex and Terminus-2, a DeepSeek-V4-Pro planner with handbook guidance produced plans preferred more often, agreed more closely with stronger-model reference plans at file and symbol level, and used fewer planner tokens. Overall preference rose from 28.3% to 38.3% on Codex and from 26.7% to 45.6% on Terminus-2. Average planner tokens fell 12.7% and 8.6%. All 24 recall, precision, and F1 comparisons against two reference models improved. This is promising evidence for behavior localization and planning. It is not evidence that final patches compile, that handbook generation is cheap, or that the method works unchanged on every repository. Adopt it as a measured navigation system: pilot on a pinned codebase, inspect coverage, compare plans, price the full lifecycle, and keep source plus tests authoritative.
A research idea becomes useful when we can answer three questions. Did the evaluation isolate the idea? Are the gains large and consistent enough to care about? Do the measurements match the claim we want to make? Harness Handbook's experiment is stronger than a screenshot demo because it compares the same planner on the same requests with one main difference: access to the behavior map. It is narrower than an end-to-end software-engineering benchmark because the agent only plans.
This final part reads the numbers carefully, explains what they do and do not establish, and turns the method into a conservative team pilot. The aim is neither hype nor dismissal. A behavior map can be valuable without being a universal solution, and a preprint can guide engineering without being treated as settled fact.
The three research questions
The paper asks whether handbook-guided localization improves plan quality while reducing planning cost; whether a weaker planner can approach substantially stronger models in implementation-site localization; and whether gains persist across request types and localization difficulty.
These questions align with the representation's intended value. If a handbook only makes explanations pleasant but does not improve localization, it has not solved the stated bottleneck. If quality rises only because the agent reads far more tokens, the gain may be an expensive context subsidy. If results disappear on cross-file or search-hostile requests, the behavior-first map is not helping where it should.
The experiment does not ask whether handbook-generated code passes tests, whether users complete tasks faster, or whether incidents decline. Those would be valuable later questions. Keeping the current claim narrow makes the reported evidence easier to interpret.
What exactly was compared?
A read-only planner built with NexAU and powered by DeepSeek-V4-Pro received a natural-language modification request. In the Baseline arm, it explored the repository directly. In the Handbook-Assisted arm, it localized code through a BGPD-consistent navigation policy using a handbook built from the same source. Requests, model, repository, permissions, and decoding settings were otherwise held constant.
Three independent models—GPT-5.5, Opus 4.8, and DeepSeek-V4-Pro—judged plan comparisons. They scored localization, scope control, and reasoning. The weighted score gave localization half the weight and each other dimension one quarter. A difference of at least three points produced a winner; smaller differences were ties.
Separate localization metrics compared predicted edit sites against independent reference plans produced by Opus 4.8 and GPT-5.5. The paper reports recall, precision, F1, and Wrong at file and symbol granularity. Wrong is the share of valid requests with zero overlap against the reference, so lower is better.
The benchmark: two harnesses, sixty requests
The evaluation uses two open-source agent harnesses. Terminus-2 is organized with function leaves because the authors had a reliable seed skeleton and a manageable function-level scope. Codex is organized with file leaves because its repository scale favors inferred stages and broad coverage.
Each harness contributes 30 behavior-driven requests, for 60 total. Requests are divided into three types. Query changes existing behavior without naming target locations. Cross-file adds an end-to-end capability spanning files or modules. Search-hostile hides relevant code behind names, mirrors, fallbacks, or rare paths that keyword search is likely to miss. Requests also receive Easy, Medium, or Hard localization labels.
This is a useful task design because it tests the paper's central translation problem. The user describes behavior rather than handing the planner a file. The cross-file and search-hostile groups probe whether a behavior map recovers relationships that local search misses.
Headline result: plans were preferred more often
On Codex, the Handbook-Assisted arm achieved a 38.3% overall win rate, compared with 28.3% for the baseline—a 10.0 percentage-point increase. On Terminus-2, it achieved 45.6% versus 26.7%, an 18.9-point increase. The remaining comparisons are ties or invalid cases under the paper's win-rate definition, so these numbers should not be read as accuracy percentages.
All three judge models favored the handbook direction on both harnesses. The gap was 10.0 points for every judge on Codex. On Terminus-2, judge-specific gaps ranged from 13.3 to 26.7 points. This consistency reduces the chance that the headline is a quirk of one evaluator.
Dimension-level results also favor the Handbook-Assisted arm on average. On Terminus-2, gains were 12.2 points for localization, 6.7 for scope control, and 4.5 for reasoning. On Codex, the corresponding gains were 2.2, 1.1, and 3.3 points. The smaller Codex dimension gaps are a reminder that results differ by repository and representation mode.
| Harness | Baseline preference | Handbook preference | Gap | Planner token change |
|---|---|---|---|---|
| Codex | 28.3% | 38.3% | +10.0 points | 0.102M → 0.089M, -12.7% |
| Terminus-2 | 26.7% | 45.6% | +18.9 points | 0.058M → 0.053M, -8.6% |
Why lower token use matters
Average planner tokens fell from about 102,000 to 89,000 per Codex request and from 58,000 to 53,000 per Terminus-2 request. The cost measure includes handbook and source context consumed during localization and plan construction.
This result supports the progressive-disclosure mechanism. The agent is not winning by ingesting a large additional manual on top of the same exploration. It is reaching relevant implementation earlier and spending less context on irrelevant files. Higher preference plus lower token use is a more persuasive combination than either alone.
However, these numbers cover planner tokens, not full lifecycle cost. Building the handbook uses static analysis and LLM-assisted generation. Maintaining it uses resynchronization. A team should measure construction, update, storage, and review costs in addition to per-request savings. A handbook pays off when repeated navigation savings and quality gains exceed that ongoing cost.
Fig 1 — Reported pairwise preference, paired with the planner-token reduction. These are win rates, not patch-pass rates.
Localization metrics: did the planner find the right sites?
The strongest table in the paper compares planned edit sites with reference plans. Across two harnesses, two reference models, two granularities, and three positive metrics, all 24 recall, precision, and F1 values improve with handbook guidance. F1 gains range from 5.0 to 18.8 percentage points.
Recall and precision rising together matters. If recall rose while precision collapsed, the planner might simply return a huge list of candidates. Instead, the results suggest it found more of the reference sites while also focusing more tightly. Wrong cases never increase and fall by as much as 25.9 points, so improvement includes fewer complete subsystem misses.
The agreement is highest on Terminus-2. With the handbook, file-level F1 reaches 84.7% against the Opus 4.8 reference and 89.3% against GPT-5.5. Symbol-level F1 reaches 77.1% and 89.3%. Precision against GPT-5.5 reaches 93.3% at both levels.
| Harness / reference / level | Baseline F1 | Handbook F1 | Gain | Wrong change |
|---|---|---|---|---|
| Codex · Opus · file | 46.6 | 61.8 | +15.2 | 37.0 → 14.8 |
| Codex · Opus · symbol | 38.3 | 57.1 | +18.8 | 44.4 → 18.5 |
| Codex · GPT-5.5 · file | 47.3 | 52.3 | +5.0 | 21.4 → 21.4 |
| Codex · GPT-5.5 · symbol | 43.8 | 51.2 | +7.4 | 28.6 → 21.4 |
| Terminus-2 · Opus · file | 74.1 | 84.7 | +10.6 | 24.1 → 13.8 |
| Terminus-2 · Opus · symbol | 64.8 | 77.1 | +12.3 | 24.1 → 13.8 |
| Terminus-2 · GPT-5.5 · file | 76.5 | 89.3 | +12.8 | 20.0 → 6.7 |
| Terminus-2 · GPT-5.5 · symbol | 73.0 | 89.3 | +16.3 | 20.0 → 6.7 |
These are macro-averaged percentages over valid requests, with missing outputs and planner errors excluded. Reference plans are model-generated rather than an unquestionable human gold standard. Agreement with two stronger models is useful, but it measures similarity to those plans, not metaphysical correctness.
Request type: gains where search should struggle
All six harness-by-request-type comparisons favor the Handbook-Assisted arm, with gains from 16.3 to 33.3 points. Codex improves most on Query requests, by 26.7 points. Terminus-2 improves most on Search-Hostile requests, by 33.3 points. Cross-file requests improve by 16.3 points on Codex and 20.0 on Terminus-2.
This pattern fits the proposed mechanism. Search-hostile cases contain mirrored, fallback, or rarely executed paths whose names may not match the request. Cross-file cases require a chain across schema, state, runtime, interface, and tests. A behavior map plus call and register relationships should help more than a file-oriented summary.
The paper does not publish enough cases to estimate fine-grained confidence for every subtype, and request labels involve judgment. A team pilot should reproduce the categories using its own historical changes rather than assuming the same distribution.
Difficulty: positive, but not a simple staircase
All six harness-by-difficulty comparisons are positive, with gaps from 3.7 to 33.3 points. Codex improves most on Easy requests; Terminus-2 improves most on Medium requests. Gains are not monotonic in the Easy, Medium, Hard labels.
This is not necessarily a problem. Difficulty labels compress many factors: number of sites, naming hostility, state coupling, architectural distance, and ambiguity. A hard request can remain hard even with a map; an easy-looking request can benefit greatly if one quiet mirror causes baseline misses.
It does mean we should avoid the slogan “the harder the task, the larger the gain.” The supported statement is more modest: gains remained positive across the labeled tiers in this dataset, and difficulty alone did not explain their size.
What is measured, inferred, and still unknown
measured The handbook-assisted planner was preferred more often on both evaluated harnesses, used fewer planner tokens, improved every reported positive localization comparison, and did not increase zero-overlap Wrong rates.
reasonable inference Better routing appears to reduce irrelevant exploration and improve both coverage and focus. This explanation is consistent with higher recall and precision plus lower token use, but the experiment does not directly instrument every navigation decision.
unknown We do not know from this paper alone how often final patches compile, pass tests, preserve behavior, or reduce human review time. We do not know the full construction and maintenance cost across repository sizes. We do not know how results transfer to dynamically typed plugin-heavy systems, monorepos with generated code, or non-agent software.
Keeping these labels separate is not academic fussiness. It prevents a planning study from becoming a claim about autonomous software delivery. The representation can be valuable at its stated layer without inheriting credit for downstream steps it did not evaluate.
Important limitations a team should carry into adoption
Two repositories are not the software world
Codex and Terminus-2 provide meaningful diversity in scale and leaf mode, but both are agent harnesses. The method should be re-evaluated on each organization's languages, runtime patterns, and repository topology.
Reference plans are produced by models
Using two strong independent references is better than one, yet both can omit sites or disagree with expert maintainers. A pilot should add human-reviewed historical diffs and known incident paths as evaluation material.
Judges are models too
Three judges and consistent directions reduce single-judge bias. They do not remove shared evaluator preferences, such as rewarding plans that sound detailed. The source-overlap metrics help anchor the result, but final engineering outcomes remain unmeasured.
Planning is not execution
A perfect edit-site list can still yield a broken patch. Implementation requires API design, concurrency reasoning, platform handling, tests, migration plans, and safe deployment. The public helper is read-only, which makes the boundary especially clear.
Generation cost is outside the headline token saving
The planner saves tokens per request, while handbook construction and resynchronization consume compute and model calls. Payback depends on change volume, repository size, cache reuse, and the cost of missed edits.
Static facts have blind spots
Reflection, generated code, dynamic plugin discovery, configuration-driven wiring, and runtime dispatch can escape supported call analysis. A coverage report is necessary but not sufficient; runtime traces and targeted tests remain valuable.
Where the approach is likely to pay off
Strong candidates are large, actively changed harnesses with distributed control flow, multiple execution modes, mirrored implementations, shared state, and expensive review. Teams that repeatedly onboard engineers or run coding agents on the same codebase can amortize construction. Safety-sensitive behavior—approval, sandboxing, secret handling, tool authorization, retries, and persistence—benefits from explicit path tracing even when a human makes the final change.
Weak candidates are tiny services, stable repositories with few changes, codebases dominated by unsupported dynamic mechanisms, or teams unable to keep the map synchronized. If one maintainer can answer every behavior question by opening two files, a multi-phase handbook may be unnecessary.
| Signal | Good candidate | Warning |
|---|---|---|
| Repository scale | Hundreds or thousands of files; repeated navigation cost | Small, obvious service |
| Behavior distribution | Cross-module state, mirrors, fallbacks, platforms | One route, one function |
| Change frequency | Frequent evolution and agent-assisted planning | Rarely modified archive |
| Parser fit | Supported languages, inspectable coverage | Heavy reflection or generated wiring |
| Governance | Pinned source, CI resync, review ownership | No owner for frozen or unmapped entries |
A four-week pilot that produces evidence
Week 1: choose a bounded target and baseline
Select one harness or subsystem with a known lifecycle. Pin a repository revision. Collect 15 to 30 historical modification requests whose final diffs and reviews are available. Include query, cross-file, and search-hostile cases. Ask maintainers to record expected files, symbols, tests, and tricky paths without seeing planner output.
Week 2: build and audit the handbook
Run static extraction first. Review language coverage, unresolved calls, generated-code boundaries, and excluded paths. Choose the leaf mode deliberately. Generate the handbook on a disposable checkout. Sample every major stage, inspect L3 locators, and test whether maintainers can navigate from a behavior question to current source.
Week 3: run an A/B planning study
Use the same planner, model, permissions, and prompts. Randomize which arm runs first. Hide final diffs from the planner. Measure file and symbol recall, precision, F1, zero-overlap misses, planner tokens, wall time, tool calls, and reviewer correction time. Ask maintainers to judge localization, scope, reasoning, and overlooked risk paths.
Week 4: test synchronization and decide
Apply several real diffs in a sandbox and run resynchronization. Inspect whether moved functions retain correct identity, changed cards refresh, ancestors update, and frozen gaps are visible. Estimate monthly construction/update cost versus navigation savings and avoided misses. Adopt only if both quality and operating cost meet a predeclared threshold.
Metrics that matter in a real rollout
Keep the paper's planning metrics, then add operational measures. Track handbook coverage by language and path. Count frozen, unmapped, and undescribed entries. Measure resynchronization latency, cache reuse, failure rate, and token cost. Track planner source reads, time to first relevant file, edit-site precision and recall against reviewed diffs, and zero-overlap subsystem misses.
For human impact, measure reviewer correction time, number of missing-site comments, rework after tests, onboarding time for behavior questions, and confidence in safety audits. For final outcomes, measure compile and test success, escaped regressions, rollback rate, and whether risky paths received appropriate approvals.
Avoid optimizing one number in isolation. A planner can reduce tokens by reading too little. It can maximize recall by naming every file. It can maximize test pass rate by making no useful change. Use a balanced scorecard tied to the workflow.
Governance: treat the handbook like a derived index
A good operational model treats the handbook like a search index or generated API reference. The repository is canonical. The handbook is rebuilt or incrementally refreshed from pinned source. CI records which commit it represents. Generated prose is reviewed through sampling and coverage checks. Frozen entries block handbook-assisted claims in their area.
Assign ownership for language adapters, stage structure, and coverage debt. Define which changes require a full rebuild. Store generation configuration and model identity. Keep secrets out of prompts and generated artifacts. If an external endpoint processes source, complete the same security and legal review required for any code-analysis service.
Version the rendered view and machine-readable synchronization state together. A page without its matching graph and repository revision is a static document, not a reliable localization tool. Expose the pinned commit in the UI so a reader can immediately judge freshness.
Security and privacy checks before generation
Source code can contain proprietary algorithms, internal hostnames, comments with incident details, test fixtures, and accidentally committed secrets. Before sending code to any model endpoint, scan and classify the repository, understand provider retention, configure an approved endpoint, and exclude data that policy forbids.
Generation workers and temporary directories also need controls. Use least-privilege credentials, disposable workspaces, encrypted storage where required, bounded logs, and cleanup policies. Do not print API keys in commands or archive them in handbook packages. A generated HTML site should not become an accidental public source browser.
The handbook can improve safety audits, but its construction creates a new information asset. Treat it accordingly. Behavior summaries can reveal sensitive architecture even when source snippets are omitted.
How to combine the Handbook with existing tools
Use a repository map for structure, semantic search for concept discovery, language-server references for exact symbols, runtime traces for actual execution, tests for behavioral proof, and the Handbook for behavior-first routing. Each artifact answers a different question.
A practical agent can begin with BGPD, then use rg, language-server queries, or code search inside selected stages. It can compare static call relationships with traces from a failing test. It can write an evidence-backed plan, execute in a sandbox, run tests, and feed the diff into resynchronization. The method is strongest as an orchestration layer over existing evidence tools.
Do not replace maintainers' architecture docs automatically. Link product intent and human decisions from relevant stages. The behavior map explains how the current implementation works; an architecture decision record explains why the team chose it and what alternatives were rejected.
A decision framework for engineering leaders
- Problem fit: Are missed or bloated edit sites a measurable source of delay or regressions?
- Evidence fit: Can static adapters cover the important languages and paths?
- Economics: Will repeated planning savings amortize construction and resync?
- Governance: Who owns coverage, frozen entries, configuration, and source privacy?
- Workflow fit: Can source verification, review, tests, and sandboxed execution remain mandatory?
- Exit criteria: What quality or cost threshold will stop the pilot?
If the problem is not behavior localization, this tool may be a distraction. If the main failures come from poor requirements, weak tests, unsafe deployment, or incorrect algorithm design, a better map will not solve them. Start from the failure mode, not from the novelty of the paper.
The deeper lesson: navigation is part of agent capability
Coding-agent discussions often center on model intelligence and patch generation. Harness Handbook highlights a less glamorous capability: finding the complete change boundary. A model can know how to implement environment merging and still fail because it edited only one of two execution paths.
Navigation is not overhead before “real reasoning.” It determines which facts enter reasoning. A behavior-centered representation can therefore improve quality without changing the planner model. The experiment's weaker planner approaching stronger reference plans is a concrete example of system design compensating for model limitations.
This fits the broader “code as agent harness” view: agent capability emerges from interfaces, state, tools, feedback, verification, and coordination around the model. Harness Handbook adds repository representation to that list. The system must make its own behavior legible enough to evolve safely.
What evidence would change this verdict?
Confidence would rise with end-to-end experiments that let the planner's evidence drive controlled patches and then measure compilation, tests, human corrections, regressions, and total cost. Multi-repository studies should include plugin-heavy Python systems, generated-code monorepos, and non-agent workflow engines. Human-authored gold localization sets would complement model references.
Confidence would fall if gains disappear after charging handbook construction and resynchronization to the same budget, if human reviewers find systematic missing paths, or if handbook organization becomes unstable across small diffs. High frozen-entry rates would indicate that alignment is not keeping pace with development. A method designed to reduce navigation debt should be judged by its own maintenance debt.
The public artifacts make these questions testable. Teams can inspect generated Codex and Terminus examples, run static extraction on their own source, package a planner skill, and compare plans. The next step is replication, not belief.
Final verdict
Harness Handbook is a strong idea with encouraging, appropriately scoped evidence. Its most valuable contribution is not an HTML manual or a new acronym. It is the explicit sequence from behavior to verified implementation, plus the rule that generated knowledge must stay aligned with current source.
The evaluation shows better plans and localization at lower planner-token cost on two real harnesses. That is enough to justify serious pilots. It is not enough to promise autonomous, regression-free harness evolution. Teams should measure final patches, total lifecycle economics, coverage debt, and human review on their own code.
If you remember one sentence from this four-part series, use this: before an agent can edit the right code, the system must make the behavior-to-code path navigable—and then verify that path against reality.
Takeaways
- The study evaluates read-only localization and plans, not final code execution.
- Handbook guidance improved overall preference by 10.0 points on Codex and 18.9 on Terminus-2 while reducing planner tokens.
- All 24 recall, precision, and F1 comparisons improved; Wrong cases never increased.
- Results were positive across request types and difficulty tiers, but not monotonic with labeled difficulty.
- Construction and maintenance cost, final patch quality, and generalization beyond two harnesses remain open adoption questions.
- A good pilot uses historical requests, human-reviewed references, A/B planning, resync tests, explicit costs, and predeclared go/no-go gates.
- Run the handbook as a derived, versioned index. Keep the repository, runtime evidence, tests, and human approval authoritative.
Frequently asked questions
Did the Handbook-Assisted planner win every case?
No. The reported overall win rates were 38.3% on Codex and 45.6% on Terminus-2, with baseline wins and ties making up the rest. The result is a higher preference rate, not universal dominance.
Does lower planner token use mean the whole system is cheaper?
Not necessarily. The reported reduction covers localization and plan construction. A business case must include initial handbook generation, resynchronization, storage, review, and model-endpoint costs.
Can a weaker model replace a stronger one with a handbook?
The study shows that a DeepSeek-V4-Pro planner's edit-site predictions moved closer to Opus 4.8 and GPT-5.5 reference plans. That does not prove equal reasoning, implementation, or final task performance.
What should a pilot use as ground truth?
Combine reviewed historical diffs, maintainer-authored expected sites, tests, incident knowledge, and independent plans. Model-generated reference plans are useful but should not be the only authority.
What is the safest first production use?
Read-only planning and review support on a bounded subsystem. Require current-source verification, expose coverage gaps, compare against human expectations, and keep execution behind existing permissions and CI.
References and further reading
This series distinguishes the authors' measured findings from practical interpretation. The paper and official artifacts below are the primary sources; the final source provides broader context for why harness code matters.
- Paper (arXiv 2607.13285v1) ↗
Wang et al., submitted July 14, 2026. The primary source for the representation, algorithms, experiment, and reported results. - Official project page ↗
The authors' visual explanation, live examples, Codex handbook, Terminus-2 handbook, and Studio walkthrough. - Official implementation ↗
Open-source generators, planner helper, resynchronization workflow, and setup instructions. - Generated Codex Handbook ↗
A large file-as-leaf example showing how the behavior map looks on a real production coding-agent repository. - Generated Terminus-2 Handbook ↗
A smaller function-as-leaf example with a supplied execution skeleton. - Code as Agent Harness ↗
Related survey that frames code as the operational substrate for reasoning, action, state, feedback, and coordination.