Outcome-oriented agent evaluation

Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents

Waseem AlShikh, Muayad Sayed Ali, Brian Kennedy, Dmytro Mozolevskyi

FIG. 01 Measure outcomes, not plumbing → Eleven metrics → Compare architectures → Across five industries
Identifier
arXiv:2511.08242
Published
Nov 11, 2025
Authors
4
Type
Paper
Area
Practical evaluations

In one paragraph

Latency and time-to-first-token say how fast an agent is. They say nothing about whether it did the job. This white paper proposes eleven outcome-based metrics, including goal completion, autonomy, resilience and business impact, and uses them to compare four agent architectures across five industries.

outcome-based, task-agnostic metrics
11
outcome-based, task-agnostic metrics
average Goal Completion Rate for the Hybrid agent architecture
88.8%
average Goal Completion Rate for the Hybrid agent architecture
architectures × domains in the simulated study
4 × 5
architectures × domains in the simulated study

FIG. 02: Mechanism

How it works

  1. Measure outcomes, not plumbing

    Infrastructure metrics such as latency and TTFT miss decision quality, autonomy and business value. The framework measures what the agent actually achieved.

  2. Eleven metrics

    They include Goal Completion Rate (GCR), Autonomy Index (AIx), Multi-Step Task Resilience (MTR) and Business Impact Efficiency (BIE). All are task-agnostic, so they work in any domain.

  3. Compare architectures

    ReAct, Chain-of-Thought, Tool-Augmented and Hybrid agents are run through a large simulated experiment.

  4. Across five industries

    Healthcare, Finance, Marketing, Legal and Customer Service, because the best agent depends on what the business values.

FIG. 03: Interactive figure

Change what you value. Watch the winner change.

Pick a domain and a priority. Eleven outcome metrics re-rank four agent architectures using the paper's own tables.

Loading interactive demo…

FIG. 04: Notes

The problem

Enterprises are deploying agents faster than they can evaluate them. Most dashboards report infrastructure metrics such as latency, throughput and time-to-first-token. Those are useful for operations, but they can’t say whether an agent made good decisions, how much human help it needed, or whether it delivered business value.

The approach

The white paper proposes eleven outcome-based, task-agnostic metrics for agent performance, including:

  • Goal Completion Rate (GCR): did the agent achieve the user’s goal?
  • Autonomy Index (AIx): how much did it get done without human intervention?
  • Multi-Step Task Resilience (MTR): does it recover when a step in a long task fails?
  • Business Impact Efficiency (BIE): how much value does it deliver per unit of cost?

We ran a large-scale simulated experiment across four agent architectures (ReAct, Chain-of-Thought, Tool-Augmented and Hybrid) and five domains (Healthcare, Finance, Marketing, Legal and Customer Service).

What we found

  • The Hybrid architecture led overall, with an average Goal Completion Rate of 88.8%. The abstract also credits it with the highest ROI. In the paper’s own Business Impact Efficiency table, though, Tool-Augmented edges it out (36.1 vs 35.8) and wins in Healthcare and Finance.
  • Rankings shift by domain and by which outcomes a business weights most. A single leaderboard number hides the trade-offs that matter.
  • The framework gives teams a common vocabulary for agent governance and procurement.

Reference

Cite this work
@misc{alshikh2025outcome,
  title={Towards Outcome-Oriented, Task-Agnostic Evaluation of AI Agents},
  author={Waseem AlShikh and Muayad Sayed Ali and Brian Kennedy and Dmytro Mozolevskyi},
  year={2025},
  eprint={2511.08242},
  archivePrefix={arXiv},
  primaryClass={cs.AI}
}