- Identifier
- WRITER Research
- Published
- Jul 31, 2024
- Authors
- 1
- Type
- Model
- Area
- Domain-specific specialization
In one paragraph
A finance-specialized LLM that scored 73% zero-shot on a CFA exam, against a human average of about 60%. Its 128k variant had the best context grounding in our FailSafeQA benchmark.
- zero-shot on a CFA exam (human average ~60%)
- 73%
- zero-shot on a CFA exam (human average ~60%)
- Context Grounding on FailSafeQA (128k variant), best of 24 models
- 0.80
- Context Grounding on FailSafeQA (128k variant), best of 24 models
FIG. 02: Mechanism
How it works
Specialize for finance
Palmyra Fin is trained for financial work such as reading filings, reasoning about markets and answering questions over long financial documents.
Test on a CFA exam
On a sample CFA exam, with basic prompting and no examples, it scored 73%. Human test takers have averaged about 60% over the past 11 years.
Test grounding on long filings
In FailSafeQA, models answer questions over 10-K filings that are perturbed, missing or swapped. The benchmark checks whether they decline when the document cannot support an answer.
Read the scores
Palmyra-Fin-128k-Instruct scored 0.80 on Context Grounding, the highest of 24 models, with 0.83 Robustness and 0.81 Compliance.
FIG. 03: Interactive figure
Answer the filing, or decline
Ask three questions about one 10-K excerpt and compare how a general model and a finance model handle a missing number and the wrong filing.
Loading interactive demo…
FIG. 04: Notes
Summary
Palmyra Fin is trained for financial analysis: reading filings, reasoning about markets and answering questions grounded in long financial documents. On an ad hoc zero-shot CFA exam test it scored 73%, well above the roughly 60% human average. It also outperformed general-purpose frontier models on an internal long-context finance evaluation.
In our FailSafeQA benchmark, Palmyra-Fin-128k-Instruct had the highest Context Grounding (0.80) of the 24 models tested. When the context couldn’t support an answer, it was the most likely to say so instead of making one up.
Reference