FailSafeQA

Expect the Unexpected: FailSafe Long Context QA for Finance

Kiran Kamble, Melisa Russak, Dmytro Mozolevskyi, Muayad Ali, Mateusz Russak, Waseem AlShikh

FIG. 01 Start from real filings → Break the question → Break the context → Judge and score
Identifier
arXiv:2502.06329
Published
Feb 10, 2025
Authors
6
Type
Benchmark
Area
Domain-specific specialization

In one paragraph

Real users misspell, ask half a question and upload the wrong document. FailSafeQA stress-tests LLMs on 10-K filings under these conditions and measures two things: whether a model stays accurate, and whether it admits when it can't answer.

best Context Grounding (Palmyra-Fin-128k-Instruct)
0.80
best Context Grounding (Palmyra-Fin-128k-Instruct)
fabrication rate for the most robust model (o3-mini)
41%
fabrication rate for the most robust model (o3-mini)
range of fabrication rates across reasoning models
41–70%
range of fabrication rates across reasoning models
off-the-shelf models evaluated
24
off-the-shelf models evaluated

FIG. 02: Mechanism

How it works

  1. Start from real filings

    Contexts come from 10-K annual reports (1998, 1999, 2017, 2018), truncated to about 25k tokens. Multi-turn QA pairs and supporting citations are generated, then filtered.

  2. Break the question

    Each query is rewritten three ways, as misspelled, incomplete and out-of-domain, to mimic how real users actually type.

  3. Break the context

    The document is degraded three ways. It is removed entirely, run through simulated OCR errors (up to 10% character error rate), or swapped for an irrelevant filing.

  4. Judge and score

    An LLM judge rates every answer. Robustness measures accuracy under query perturbations. Context Grounding measures whether the model declines when the context cannot support an answer. Compliance combines the two.

FIG. 03: Interactive figure

Break the question. Break the document.

Corrupt the query and the context, see what a compliant model should do, then move β to trade robustness against grounding.

Loading interactive demo…

FIG. 04: Notes

The problem

Benchmarks usually assume a clean question and the right document. Production rarely provides either. In finance, a model that confidently answers from the wrong 10-K is worse than one that says “I can’t find that.”

The approach

FailSafeQA takes long 10-K contexts and applies six controlled perturbations:

  • Query perturbations: misspelled, incomplete, out-of-domain.
  • Context perturbations: missing context, OCR-degraded context, irrelevant context.

An LLM judge (Qwen2.5-72B-Instruct) rates each answer on a 1–6 scale. Answers rated 4 or higher count as compliant.

Each model gets three scores:

  • Robustness: whether the model still answers correctly when the question is degraded.
  • Context Grounding: whether the model refuses or flags the problem when the context can’t support an answer, instead of hallucinating.
  • LLM Compliance: an F-score of the two with β = 0.5, which weights grounding more heavily.

What we found

There is a clear trade-off between robustness and grounding.

  • OpenAI o3-mini was the most robust (0.90 against a 0.98 baseline), yet it fabricated information in 41% of the cases where it should have declined.
  • Palmyra-Fin-128k-Instruct had the best grounding (0.80), with Robustness 0.83 and Compliance 0.81. It still failed to stay robust in 17% of cases.
  • Across reasoning models, fabrication rates ranged from 41% to 70%.

The dataset is open: 220 examples on Hugging Face under the MIT license.

Reference

Cite this work
@misc{kamble2025expect,
  title={Expect the Unexpected: FailSafe Long Context QA for Finance},
  author={Kiran Kamble and Melisa Russak and Dmytro Mozolevskyi and Muayad Ali and Mateusz Russak and Waseem AlShikh},
  year={2025},
  eprint={2502.06329},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}