Outcome-oriented agent evaluation
Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents
- Identifier
- arXiv:2511.08242
- Published
- Nov 11, 2025
- Authors
- 4
- Type
- Paper
- Area
- Practical evaluations
In one paragraph
Latency and time-to-first-token say how fast an agent is. They say nothing about whether it did the job. This white paper proposes eleven outcome-based metrics, including goal completion, autonomy, resilience and business impact, and uses them to compare four agent architectures across five industries.
- outcome-based, task-agnostic metrics
- 11
- outcome-based, task-agnostic metrics
- average Goal Completion Rate for the Hybrid agent architecture
- 88.8%
- average Goal Completion Rate for the Hybrid agent architecture
- architectures × domains in the simulated study
- 4 × 5
- architectures × domains in the simulated study
FIG. 02: Mechanism
How it works
Measure outcomes, not plumbing
Infrastructure metrics such as latency and TTFT miss decision quality, autonomy and business value. The framework measures what the agent actually achieved.
Eleven metrics
They include Goal Completion Rate (GCR), Autonomy Index (AIx), Multi-Step Task Resilience (MTR) and Business Impact Efficiency (BIE). All are task-agnostic, so they work in any domain.
Compare architectures
ReAct, Chain-of-Thought, Tool-Augmented and Hybrid agents are run through a large simulated experiment.
Across five industries
Healthcare, Finance, Marketing, Legal and Customer Service, because the best agent depends on what the business values.
FIG. 03: Interactive figure
Change what you value. Watch the winner change.
Pick a domain and a priority. Eleven outcome metrics re-rank four agent architectures using the paper's own tables.
Loading interactive demo…
FIG. 04: Notes
The problem
Enterprises are deploying agents faster than they can evaluate them. Most dashboards report infrastructure metrics such as latency, throughput and time-to-first-token. Those are useful for operations, but they can’t say whether an agent made good decisions, how much human help it needed, or whether it delivered business value.
The approach
The white paper proposes eleven outcome-based, task-agnostic metrics for agent performance, including:
- Goal Completion Rate (GCR): did the agent achieve the user’s goal?
- Autonomy Index (AIx): how much did it get done without human intervention?
- Multi-Step Task Resilience (MTR): does it recover when a step in a long task fails?
- Business Impact Efficiency (BIE): how much value does it deliver per unit of cost?
We ran a large-scale simulated experiment across four agent architectures (ReAct, Chain-of-Thought, Tool-Augmented and Hybrid) and five domains (Healthcare, Finance, Marketing, Legal and Customer Service).
What we found
- The Hybrid architecture led overall, with an average Goal Completion Rate of 88.8%. The abstract also credits it with the highest ROI. In the paper’s own Business Impact Efficiency table, though, Tool-Augmented edges it out (36.1 vs 35.8) and wins in Healthcare and Finance.
- Rankings shift by domain and by which outcomes a business weights most. A single leaderboard number hides the trade-offs that matter.
- The framework gives teams a common vocabulary for agent governance and procurement.
Reference
@misc{alshikh2025outcome,
title={Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents},
author={Waseem AlShikh and Muayad Sayed Ali and Brian Kennedy and Dmytro Mozolevskyi},
year={2025},
eprint={2511.08242},
archivePrefix={arXiv},
primaryClass={cs.AI}
}