Writing in the Margins
Writing in the Margins: Better Inference Pattern for Long Context Retrieval
- Identifier
- arXiv:2408.14906
- Published
- Aug 27, 2024
- Authors
- 7
- Type
- Paper
- Area
- Enterprise-optimized models
In one paragraph
Long documents are processed in chunks, and the model writes short notes in the margin as it reads each one. The relevant notes are carried into the final answer. Off-the-shelf models handle long context better, with no fine-tuning.
- average accuracy on multi-hop reasoning (HotpotQA, MultiHop-RAG)
- +7.5%
- average accuracy on multi-hop reasoning (HotpotQA, MultiHop-RAG)
- F1 increase on aggregation tasks (CWE)
- >30%
- F1 increase on aggregation tasks (CWE)
- fine-tuning steps; works with off-the-shelf models
- 0
- fine-tuning steps; works with off-the-shelf models
FIG. 02: Mechanism
How it works
Chunk the context
The long input is split into segments, and the model's KV cache is filled one segment at a time (chunked prefill), just as it would be anyway.
Write in the margins
After each segment is prefilled, the model generates a short "margin", a note on what that segment says about the question.
Classify the notes
Each margin is classified as relevant or irrelevant to the query, and only the useful notes are kept.
Answer with notes in hand
The relevant margins are added to the end of the prompt, so the final answer draws on both the full context and the distilled notes.
FIG. 03: Interactive figure
Read a long report the way the model does.
Prefill the document chunk by chunk, watch margin notes appear, then switch notes off and see how the final prompt changes.
Loading interactive demo…
FIG. 04: Notes
The problem
Models with long context windows still get “lost in the middle”. As the input grows, the relevant facts are drowned out by everything around them. Retrieval (RAG) helps, but it can miss what matters when the answer has to be pieced together across many parts of a document.
The approach
Writing in the Margins (WiM) uses the chunked prefill that long-context inference already performs. As each chunk enters the KV cache, the model writes a short margin note about that chunk. The notes are classified for relevance, and the useful ones are added to the end of the prompt before the final answer. The model reads the whole document and keeps notes as it goes, much as a careful human reader would.
The pattern adds little compute overhead and needs no fine-tuning. Because margins arrive while reading is still under way, they can also be shown to users as progress.
What we found
- +7.5% average accuracy on reasoning tasks (HotpotQA, MultiHop-RAG).
- More than 30% F1 improvement on aggregation tasks (CWE).
- Works across off-the-shelf models, compared against both long-context and RAG baselines.
Reference
@misc{russak2024writing,
title={Writing in the Margins: Better Inference Pattern for Long Context Retrieval},
author={Melisa Russak and Umar Jamil and Christopher Bryant and Kiran Kamble and Axel Magnuson and Mateusz Russak and Waseem AlShikh},
year={2024},
eprint={2408.14906},
archivePrefix={arXiv},
primaryClass={cs.CL}
}