- Identifier
- arXiv:2502.06329
- Published
- Feb 10, 2025
- Authors
- 6
- Type
- Benchmark
- Area
- Domain-specific specialization
In one paragraph
Real users misspell, ask half a question and upload the wrong document. FailSafeQA stress-tests LLMs on 10-K filings under these conditions and measures two things: whether a model stays accurate, and whether it admits when it can't answer.
- best Context Grounding (Palmyra-Fin-128k-Instruct)
- 0.80
- best Context Grounding (Palmyra-Fin-128k-Instruct)
- fabrication rate for the most robust model (o3-mini)
- 41%
- fabrication rate for the most robust model (o3-mini)
- range of fabrication rates across reasoning models
- 41–70%
- range of fabrication rates across reasoning models
- off-the-shelf models evaluated
- 24
- off-the-shelf models evaluated
FIG. 02: Mechanism
How it works
Start from real filings
Contexts come from 10-K annual reports (1998, 1999, 2017, 2018), truncated to about 25k tokens. Multi-turn QA pairs and supporting citations are generated, then filtered.
Break the question
Each query is rewritten three ways, as misspelled, incomplete and out-of-domain, to mimic how real users actually type.
Break the context
The document is degraded three ways. It is removed entirely, run through simulated OCR errors (up to 10% character error rate), or swapped for an irrelevant filing.
Judge and score
An LLM judge rates every answer. Robustness measures accuracy under query perturbations. Context Grounding measures whether the model declines when the context cannot support an answer. Compliance combines the two.
FIG. 03: Interactive figure
Break the question. Break the document.
Corrupt the query and the context, see what a compliant model should do, then move β to trade robustness against grounding.
Loading interactive demo…
FIG. 04: Notes
The problem
Benchmarks usually assume a clean question and the right document. Production rarely provides either. In finance, a model that confidently answers from the wrong 10-K is worse than one that says “I can’t find that.”
The approach
FailSafeQA takes long 10-K contexts and applies six controlled perturbations:
- Query perturbations: misspelled, incomplete, out-of-domain.
- Context perturbations: missing context, OCR-degraded context, irrelevant context.
An LLM judge (Qwen2.5-72B-Instruct) rates each answer on a 1–6 scale. Answers rated 4 or higher count as compliant.
Each model gets three scores:
- Robustness: whether the model still answers correctly when the question is degraded.
- Context Grounding: whether the model refuses or flags the problem when the context can’t support an answer, instead of hallucinating.
- LLM Compliance: an F-score of the two with β = 0.5, which weights grounding more heavily.
What we found
There is a clear trade-off between robustness and grounding.
- OpenAI o3-mini was the most robust (0.90 against a 0.98 baseline), yet it fabricated information in 41% of the cases where it should have declined.
- Palmyra-Fin-128k-Instruct had the best grounding (0.80), with Robustness 0.83 and Compliance 0.81. It still failed to stay robust in 17% of cases.
- Across reasoning models, fabrication rates ranged from 41% to 70%.
The dataset is open: 220 examples on Hugging Face under the MIT license.
Reference
@misc{kamble2025expect,
title={Expect the Unexpected: FailSafe Long Context QA for Finance},
author={Kiran Kamble and Melisa Russak and Dmytro Mozolevskyi and Muayad Ali and Mateusz Russak and Waseem AlShikh},
year={2025},
eprint={2502.06329},
archivePrefix={arXiv},
primaryClass={cs.CL}
}