Prediction ≠ prevention
Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
- Identifier
- arXiv:2602.03338
- Published
- Feb 3, 2026
- Authors
- 4
- Type
- Paper
- Area
- Practical evaluations
In one paragraph
A critic that predicts agent failures with 0.94 AUROC sounds like a safety net. Put it in charge of intervening, though, and it can cut success by 26 points. Intervening rescues some failing runs and breaks some that would have succeeded. We show how to measure that trade-off before you deploy.
- critic AUROC (an accurate failure predictor)
- 0.94
- critic AUROC (an accurate failure predictor)
- worst success-rate collapse after enabling intervention
- −26 pp
- worst success-rate collapse after enabling intervention
- gain on high-failure ALFWorld (p = 0.014)
- +2.8 pp
- gain on high-failure ALFWorld (p = 0.014)
- pilot tasks needed to predict whether intervention will help
- 50
- pilot tasks needed to predict whether intervention will help
FIG. 02: Mechanism
How it works
Train a critic
A binary LLM critic watches an agent's trajectory and predicts whether it is heading toward failure. Offline it scores well, with AUROC 0.94.
Let it intervene
When the critic's failure score crosses a threshold, it steps in mid-trajectory to steer the agent.
Count both effects
Some interventions rescue runs that would have failed (recovery). Others derail runs that would have succeeded (disruption).
Know which regime you're in
On tasks where the agent usually succeeds, disruption dominates. On tasks where it usually fails, recovery can win. Accuracy alone can't tell you which case you're in.
Pilot before you deploy
A 50-task pilot of the agent with and without the critic shows whether intervention will help or hurt before it reaches production.
FIG. 03: Interactive figure
An accurate critic. A worse agent.
Move the intervention threshold, count rescued and disrupted runs, then run the 50-task pilot before you deploy.
Loading interactive demo…
FIG. 04: Notes
The problem
A popular safety pattern for AI agents is to put a critic model beside the agent that predicts failures and intervenes before they happen. The reasoning seems sound: if the critic predicts failures accurately, letting it intervene should prevent them.
The approach
We trained a binary LLM critic that reaches AUROC 0.94 at predicting agent failure, then let it intervene during live runs across several models and benchmarks. For each task, we tracked whether the intervention rescued a run that was failing or disrupted one that would have succeeded.
What we found
- Accuracy does not equal benefit. The same critic caused a 26 percentage point collapse on one model and almost no change on another.
- The disruption–recovery trade-off. Intervening rescues some failing trajectories but disrupts others that would have succeeded. On tasks with high success rates, disruption wins, and intervention degraded performance by 0 to −26 pp.
- Where it helps. On the high-failure ALFWorld benchmark, intervention gave a modest but significant +2.8 pp (p = 0.014).
- A practical test. A 50-task pilot run before deployment reliably predicts whether intervention will help or hurt for a given agent and workload.
Reference
@misc{vasudev2026accurate,
title={Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention},
author={Rakshith Vasudev and Melisa Russak and Dan Bikel and Waseem AlShikh},
year={2026},
eprint={2602.03338},
archivePrefix={arXiv},
primaryClass={cs.CL}
}