Palmyra Fin

Palmyra Fin: A domain-specialized LLM for finance

Sam Julien

FIG. 01 Specialize for finance → Test on a CFA exam → Test grounding on long filings → Read the scores
Identifier
WRITER Research
Published
Jul 31, 2024
Authors
1
Type
Model
Area
Domain-specific specialization

In one paragraph

A finance-specialized LLM that scored 73% zero-shot on a CFA exam, against a human average of about 60%. Its 128k variant had the best context grounding in our FailSafeQA benchmark.

zero-shot on a CFA exam (human average ~60%)
73%
zero-shot on a CFA exam (human average ~60%)
Context Grounding on FailSafeQA (128k variant), best of 24 models
0.80
Context Grounding on FailSafeQA (128k variant), best of 24 models

FIG. 02: Mechanism

How it works

  1. Specialize for finance

    Palmyra Fin is trained for financial work such as reading filings, reasoning about markets and answering questions over long financial documents.

  2. Test on a CFA exam

    On a sample CFA exam, with basic prompting and no examples, it scored 73%. Human test takers have averaged about 60% over the past 11 years.

  3. Test grounding on long filings

    In FailSafeQA, models answer questions over 10-K filings that are perturbed, missing or swapped. The benchmark checks whether they decline when the document cannot support an answer.

  4. Read the scores

    Palmyra-Fin-128k-Instruct scored 0.80 on Context Grounding, the highest of 24 models, with 0.83 Robustness and 0.81 Compliance.

FIG. 03: Interactive figure

Answer the filing, or decline

Ask three questions about one 10-K excerpt and compare how a general model and a finance model handle a missing number and the wrong filing.

Loading interactive demo…

FIG. 04: Notes

Summary

Palmyra Fin is trained for financial analysis: reading filings, reasoning about markets and answering questions grounded in long financial documents. On an ad hoc zero-shot CFA exam test it scored 73%, well above the roughly 60% human average. It also outperformed general-purpose frontier models on an internal long-context finance evaluation.

In our FailSafeQA benchmark, Palmyra-Fin-128k-Instruct had the highest Context Grounding (0.80) of the 24 models tested. When the context couldn’t support an answer, it was the most likely to say so instead of making one up.

Reference