Your agent's recovery step helps some runs and wrecks others. A paper shows how to measure that
A 30 September paper treats agent error recovery as a per-run causal decision: fork from the same state with and without a refresh, count rescues and harms, and route recovery only where it pays.
On this page
Most agent harnesses have a reflex. Something looks off, so re-read the state, re-query the tool, replan. It feels like free insurance.
It isn’t free, and the usual way of checking it hides the bill. You turn the reflex on, average task success goes up a point or two, you ship it. Meanwhile the reflex may be rescuing some runs and quietly breaking others that would have finished fine. An average can’t tell those apart.
A paper from 30 September, When Harnesses Lose the Signal: Causal Evaluation of Recovery in LLM Agents (Xiao et al., arXiv:2610.00372), makes that bill visible.
The problem
Recovery is usually judged by mean success with it on versus off. That compares two populations. It never asks the question you actually care about for a single run: given this exact state, would the refresh have changed the outcome, and which way?
The same refresh can rescue a run that was heading for failure and derail one that was heading for success. Those effects cancel in the average. A net gain of +3 points could be 11 rescues and 2 harms, or 40 rescues and 37 harms. Those are very different products to operate.
The core idea
Treat recovery as a decision made at a state, and measure it with paired counterfactuals.
- Run the agent to some checkpoint.
- Fork from that identical state: one branch continues untouched, the other applies the recovery step (here, a “refresh”, an extra query to the environment).
- Compare outcomes and sort each pair into rescue, harm or unchanged.
Because both branches share everything up to the fork, the difference is attributable to the intervention, not to which tasks happened to land in which arm. The authors also vary the observation condition (clean, stale, missing) and the delay before recovery (0, 1, 2 or 4 actions).
Why it works as a method
It turns an aggregate into a table you can reason about. Rescue rate and harm rate are separate numbers, and each depends on conditions you can often detect. A refresh on a clean observation has nothing to fix and a lot to disturb. A refresh after a stale observation has something real to repair.
That’s the whole insight, and it’s an old one from experimentation practice: estimate the effect per unit, then ask where the effect is positive.
The router
Once you know recovery helps in some states and hurts in others, the obvious step is a policy that fires it selectively. The authors call theirs the Causal Intervention Router (CIR).
As described in a third-party write-up of the paper (I could verify the headline numbers against the arXiv abstract, the rest against that summary only, so check the paper before quoting): CIR uses only pre-decision features, fits small logistic models for observation error, success without recovery, and success after refresh, then intervenes at the first qualifying checkpoint, at most once per trajectory, under a cap on harm to clean trajectories.
The cap is the design choice worth stealing. “Don’t make good runs worse” is a constraint, not a hope.
Results, with the caveats attached
Setup: ALFWorld household tasks, Qwen3-14B, a ReAct-style harness, greedy decoding.
- The abstract reports CIR lifting success from 70.33% to 73.33%, a gain of 3.00 points, on long-horizon tasks.
- Per the write-up: unconditional refresh hurt clean-observation runs (about −6 points on the factual-success subset) and helped two-step-stale ones. On the held-out test set CIR logged 11 rescues against 2 harms, left clean episodes unchanged, and intervened in about 16% of episodes.
- A mechanism control matters: a refresh that queries the environment but hides the returned text still recovered most of the gain, and a replan-only variant barely moved. So part of the benefit comes from interrupting the execution stream, not from the new information.
Now the honest part. One environment, one model, a prefix-feasible task subset of about a hundred tasks, a 300-episode test set. Several confidence intervals cross zero, and simple anomaly-based baselines landed within about one point of CIR. Thresholds are tuned to this setup. Treat the measurement protocol as the contribution and the +3 as a demo, not a number to plan around.
Architectural implications
- Log checkpoints you can fork from. Paired evaluation needs resumable state: messages, tool results, environment handle. If your agent state lives half in a process and half in a remote system, you can’t replay it. That’s a durability requirement hiding inside an eval method.
- Separate “better content” from “a pause”. If a content-free refresh helps, a cheaper nudge (a replan prompt, a delay) might capture some of the gain at a fraction of the tool cost. Test the ablation before building the router.
- Make recovery a policy, not a hardcoded branch. Anything that fires on “looks wrong” should expose its trigger, its cap and its outcome label to telemetry.
- Budget the evaluation. Every pair doubles rollouts at the fork. Do it on a sampled set of checkpoints, not every step.
Where I’d be skeptical
Counterfactual forks assume the environment can be reset to the same state and that sampling noise is controlled. Greedy decoding gives that here. At temperature above zero you need several forks per checkpoint, and the paired counts get noisy fast. With small samples, rescue-versus-harm counts like 11 versus 2 are encouraging but fragile.
Also, ALFWorld is a household simulator with a clean notion of “stale observation”. Real agents fail in messier ways: a flaky API returned partial data, a file changed under you. Defining what counts as a corrupted state in your system is most of the work.
The one thing worth keeping
Report recovery as three numbers, not one: rescued, harmed, unchanged. If you only log the net, you can’t tell a good policy from a lucky mix.
No official code link turned up for this paper when I looked, so the exercise below builds the protocol from scratch.
Try it in 38 minutes
You need any agent with replayable traces (even a toy tool-calling loop over a mock API).
- Pick a recovery behaviour you already have: retry, re-read, replan.
- Take 30 saved traces and choose a checkpoint in each, say the first tool error or a random mid-point.
- For each, run two continuations from the identical state: recovery on, recovery off. Use temperature 0 or three samples each and take the majority.
- Label each pair rescue, harm or unchanged. Write the three counts and the net.
- Split by a cheap feature you can see before deciding (did the last tool return an error? how many steps in?) and look for a slice where harms outnumber rescues.
If one slice shows harm, you’ve found a guard condition for your harness. If the table is all “unchanged”, the reflex may just be costing you tokens.
Ship the reflex when the table says so, not when the average does. And if the experiment sits half-built on your disk this week, that’s fine; the traces will still be there next week.
Researched and drafted with AI, checked against the linked sources. How we use AI →