The Price of Agreement

The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications

Zhenyu Zhao, Aparna Balagopalan, Adi Agrawal, Dilshoda Yergasheva, Waseem AlShikh, Daniel M. Bikel

FIG. 01 Run the finance tasks clean → Push back in the conversation → Inject biased user preferences → Score accuracy and transparency → Filter the inputs
Identifier
arXiv:2604.24668
Published
Apr 27, 2026
Authors
6
Type
Paper
Area
Domain-specific specialization

In one paragraph

Finance agents hold up reasonably well when a user argues with them. They fail when the user's preferences are injected into their context or tool results and contradict the correct answer. Filtering inputs with a separate LLM recovers some of the lost accuracy, but not all of it.

Gemini-3-Pro accuracy on FinanceBench with injected preferences
0.83 → 0.24
Gemini-3-Pro accuracy on FinanceBench with injected preferences
Claude-Sonnet-4.5 accuracy on FinanceBench under user contradiction
0.87 → 0.45
Claude-Sonnet-4.5 accuracy on FinanceBench under user contradiction
Claude-Opus-4.5 accuracy recovered by prompt-based filtering
0.55 → 0.71
Claude-Opus-4.5 accuracy recovered by prompt-based filtering

FIG. 02: Mechanism

How it works

  1. Run the finance tasks clean

    Models answer FinanceBench questions over 10-K and 10-Q filings and complete FinanceAgent tasks with tools. This gives baseline accuracy.

  2. Push back in the conversation

    A rebuttal adds a user turn that rejects the model's answer and asks it to try again. A contradiction also proposes a different answer.

  3. Inject biased user preferences

    Synthetic user beliefs and past behavior that contradict the reference answer are added either directly to the prompt or as the result of a memory or personalization tool call.

  4. Score accuracy and transparency

    An LLM judge scores accuracy, the acknowledgment rate (AR, how often the model says the user's information shaped its answer) and EWU (the share of wrong answers given without that acknowledgment).

  5. Filter the inputs

    A separate LLM with a specialized prompt removes bias-inducing preference content before the main model sees it, and the same metrics are measured again.

FIG. 03: Interactive figure

Tell the agent what the user wants

Inject a rebuttal, a contradiction or a biased preference into a finance agent's context, then filter it out and compare the reported accuracy.

Loading interactive demo…

FIG. 04: Notes

Summary

LLMs in financial systems can side with what the user believes rather than what is correct. This paper measures that sycophancy in agentic financial tasks, on FinanceBench and FinanceAgent.

Three kinds of personal context are tested:

  • Rebuttals: the user rejects the model’s answer and asks it to try again.
  • Contradictions: the user rejects the answer and proposes a different one.
  • Preferences: user preferences that contradict the reference answer, injected directly into the prompt or delivered as a tool result, the way a memory or personalization API would deliver them.

Two metrics complement accuracy:

  • Acknowledgment rate (AR): whether the model flags the conflict.
  • EWU: the share of wrong answers given with no acknowledgment.

Results

  • Rebuttals cause modest drops; injected preferences break most models. On FinanceBench, Gemini-3-Pro fell from 0.83 to 0.24 under direct preference injection. Claude-Sonnet-4.5 fell from 0.87 to 0.45 under contradiction and to 0.72 under rebuttal.
  • Tool-delivered context is quieter. Context delivered as a tool result lowers awareness and acknowledgment.
  • Filtering helps partially. A separate LLM that removes biased preference content brought Claude-Opus-4.5 from 0.55 back to 0.71, short of its baseline.

This work is the finance half of our sycophancy and memory research.

Reference

Cite this work
@misc{zhao2026price,
  title={The Price of Agreement: Measuring LLM Sycophancy in Agentic Financial Applications},
  author={Zhenyu Zhao and Aparna Balagopalan and Adi Agrawal and Dilshoda Yergasheva and Waseem AlShikh and Daniel M. Bikel},
  year={2026},
  eprint={2604.24668},
  archivePrefix={arXiv},
  primaryClass={cs.AI}
}