Durability, recovery & orchestration.

An agent that runs for an hour will hit a crash, a timeout, a rate limit, or a machine restart. A toy harness loses everything and starts over; a real one checkpoints every model turn and tool call so it can replay from where it left off. Module 4 makes your harness durable: checkpointing and replay, self-healing loops (retries, failure classification, resumable sessions), sub-agents for when one context can't hold the job, and human-in-the-loop supervision (plans, approvals, escalation). You build checkpointed execution and a sub-agent dispatcher with an approval gate.

Modules 1–3 built an agent that acts, stays safe, and survives long sessions. But it's still fragile in a way that only shows up in production: it has no memory of the run itself. Kill the process at turn 40 of a 60-turn refactor and everything is gone — the plan, the decisions, the half-finished edits' logical thread. For an interactive session that's annoying; for a long autonomous task, or one running unattended overnight, it's fatal. This module is about making a run survive the world being unreliable.

Durable execution: checkpoint every turn

The core idea comes from durable-workflow systems (Temporal is the canonical reference): persist enough state after every step that you can reconstruct the run and continue. For an agent harness, the "steps" are exactly the loop's events — each model turn and each tool call. If you write those to durable storage as they happen, a crash becomes a pause, not a loss.

Two flavors of durability, and it's worth knowing which you need:

• State snapshotting. After each event, write the current message array (and any run metadata — turn count, budget spent, plan) to durable storage. On restart, load the latest snapshot and continue the loop. Simple, and enough for most single-machine harnesses. This is essentially the Module 3 message array, persisted.
• Event-sourced replay. Record each event as an immutable entry in an append-only log; reconstruct state by replaying the log from the start. More powerful — you get a full audit trail, time-travel debugging, and the ability to replay with a fixed tool result — but heavier. This is what Temporal-style durable execution and Hermes's SQLite session plane give you.

Self-healing loops: retries and failure classification

Checkpointing handles hard crashes. Most failures are softer — a transient network blip, a rate limit, a flaky tool — and a good harness absorbs them without a human noticing. The key is failure classification: not all errors deserve the same response.

The distinction that trips people up is transient vs agent-correctable. A 429 is the harness's problem — the model can't fix a rate limit, so the harness silently backs off and retries without ever bothering the model or the human. A failed test is the model's problem — the harness feeds it back as an observation and lets the model reason about the fix. Sending a 429 to the model wastes tokens on something it can't act on; hiding a failed test from the model prevents it from doing its job. Route each failure to whoever can actually resolve it.

"Self-healing" is the sum of these: transient errors retried automatically, agent-correctable errors fed back, fatal errors escalated, and — underneath it all — checkpointing so that even a total crash resumes rather than restarts. A well-built harness can run for hours across rate limits, flaky tools, and a machine reboot, and finish the job. That resilience is a defining feature of a production harness versus a demo.

Sub-agents and handoffs

Sometimes one context can't hold the job. A task with a big, separable sub-part — "analyze all 200 files and summarize the auth flow," "research these ten libraries" — would blow the main agent's context budget (Module 3) with detail the main agent doesn't need. The answer is a sub-agent: spawn a fresh agent with its own clean context window, hand it the sub-task, let it run its own loop, and return only a short result to the parent.

Sub-agents buy three things: context isolation (the sub-task's token churn never touches the parent's window — the "isolation" tactic from Module 3), parallelism (independent sub-agents run concurrently, cutting wall-clock time), and specialization (a sub-agent can have a different system prompt, tool set, or even model — a cheap model for search, an expensive one for reasoning). A handoff is the related pattern where control passes from one agent to another specialist (a "planner" hands to a "coder") rather than nesting.

Supervision and human-in-the-loop

Autonomy is a spectrum, and durability lets you place a human at exactly the points that matter. Three supervision patterns, in increasing autonomy:

• Plans. Before executing, the agent produces a plan and a human approves it. This front-loads oversight: you review the intent once, cheaply, instead of every action. A checkpointed plan also becomes the agent's north star — it can be re-injected after compaction so the agent doesn't drift from what was approved.
• Approvals. The per-action gate from Module 2, now placed strategically: auto-allow the safe and reversible, require approval for the risky and irreversible. Durability makes this clean — the harness checkpoints, blocks on the approval, and resumes when the human answers (even if that's hours later, from a different device).
• Escalation. When the agent is stuck (circuit breaker tripped), uncertain, or facing a fatal error, it escalates to a human rather than guessing or spinning. Good escalation is specific: "I've tried X and Y, both failed with Z, I need a decision on W" — not a silent stall. Escalation is the safety valve that makes unattended autonomy acceptable.

The unifying idea: a human is a special kind of tool the agent can call — one that's slow, expensive, and authoritative. Supervision engineering is deciding when the agent must call that tool. Too often and you've built a very expensive autocomplete; too rarely and you've built a liability. The right placement — plans up front, approvals on the irreversible, escalation on the stuck — is what lets a harness run mostly-autonomously while staying safe.

You build — checkpointed execution + a sub-agent dispatcher

import json, pathlib, time

LOG = pathlib.Path(".harness/checkpoint.jsonl")
LOG.parent.mkdir(exist_ok=True)

def checkpoint(state):                        # append-only durable log
    with LOG.open("a") as f:
        f.write(json.dumps({"t": time.time(), **state}) + "\n")

def resume():                                 # rebuild state from the last checkpoint
    if not LOG.exists(): return None
    last = LOG.read_text().strip().splitlines()[-1]
    return json.loads(last)

def durable_run(client, goal, max_turns=200):
    prior = resume()
    messages = prior["messages"] if prior else [{"role": "user", "content": goal}]
    start = prior["turn"] if prior else 0
    for turn in range(start, max_turns):
        resp = call_with_retries(client, messages)         # self-healing (below)
        messages.append({"role": "assistant", "content": resp.content})
        calls = [b for b in resp.content if b.type == "tool_use"]
        if not calls:
            checkpoint({"turn": turn, "messages": messages, "done": True}); return resp
        results = [{"type": "tool_result", "tool_use_id": c.id,
                    "content": str(run_tool(c.name, c.input))} for c in calls]
        messages.append({"role": "user", "content": results})
        checkpoint({"turn": turn + 1, "messages": messages})   # persist AFTER each turn

def call_with_retries(client, messages, tries=5):            # failure classification
    for i in range(tries):
        try:
            return client.messages.create(model="claude-sonnet-5", max_tokens=2048,
                                          tools=TOOLS, messages=messages)
        except RateLimitError:                               # transient -> back off
            time.sleep((2 ** i) + random.random())
        except AuthenticationError:                          # fatal -> stop
            raise
    raise RuntimeError("exhausted retries")

# ---- sub-agent dispatcher (a tool the model can call) ----
def spawn_subagent(task, system=None):
    if input(f"\n  ⚠ spawn sub-agent for: {task}\n  approve? [y/N] ").lower() != "y":
        return "DENIED"
    child = Anthropic()                                      # fresh client = fresh context
    result = child.messages.create(                          # its own isolated loop (elided)
        model="claude-sonnet-5", max_tokens=2048,
        system=system or "You are a focused sub-agent. Do the task, return a SHORT summary.",
        tools=TOOLS, messages=[{"role": "user", "content": task}])
    return summarize(result)                                 # only the summary returns

Everything in this module is here. checkpoint/resume give durability — kill the process mid-run and durable_run picks up at the last persisted turn. call_with_retries classifies failures — back off on transient, stop on fatal (agent-correctable errors are already handled inside run_tool from Module 2). And spawn_subagent is a tool the model can invoke to delegate a heavy sub-task to an isolated child context behind an approval gate, returning only a summary — context isolation and human-in-the-loop in one function.

FAQ

Takeaways

• Durable execution: checkpoint after every turn and tool call so a crash becomes a replay, not a loss. Snapshot state, or event-source for full replay/audit.
• Replay + side effects needs idempotency keys, or one crash becomes a duplicated action.
• Self-healing = classify failures: retry transient (backoff+jitter), escalate fatal, feed agent-correctable back, circuit-break loop pathologies.
• Sub-agents isolate context, enable parallelism, and allow specialization — but cost ~15× tokens when fanned out, so spend them deliberately.
• Supervision: plans (approve intent up front), approvals (gate the irreversible), escalation (call the human when stuck). A human is a slow, authoritative tool.
• Durability turns an agent from a thing you babysit into a long-lived, resumable, externally-steerable process (Hermes's model).

References & further reading

• Temporal — Durable execution for AI agents — checkpointing, replay, and idempotency for agent workflows.
• Arize — Hermes harness architecture — sessions-as-infrastructure, resumable long-running agents.
• Anthropic — Building effective agents — orchestrator-workers, sub-agents, and human-in-the-loop patterns.
• Loop engineering — failure classification, retries, circuit breakers, the termination doctrine.
• LangChain — Anatomy of an Agent Harness — subagent spawning and verification hooks.