Writing in the Margins

Writing in the Margins: Better Inference Pattern for Long Context Retrieval

Melisa Russak, Umar Jamil, Christopher Bryant, Kiran Kamble, Axel Magnuson, Mateusz Russak, Waseem AlShikh

FIG. 01 Chunk the context → Write in the margins → Classify the notes → Answer with notes in hand
Identifier
arXiv:2408.14906
Published
Aug 27, 2024
Authors
7
Type
Paper
Area
Enterprise-optimized models

In one paragraph

Long documents are processed in chunks, and the model writes short notes in the margin as it reads each one. The relevant notes are carried into the final answer. Off-the-shelf models handle long context better, with no fine-tuning.

average accuracy on multi-hop reasoning (HotpotQA, MultiHop-RAG)
+7.5%
average accuracy on multi-hop reasoning (HotpotQA, MultiHop-RAG)
F1 increase on aggregation tasks (CWE)
>30%
F1 increase on aggregation tasks (CWE)
fine-tuning steps; works with off-the-shelf models
0
fine-tuning steps; works with off-the-shelf models

FIG. 02: Mechanism

How it works

  1. Chunk the context

    The long input is split into segments, and the model's KV cache is filled one segment at a time (chunked prefill), just as it would be anyway.

  2. Write in the margins

    After each segment is prefilled, the model generates a short "margin", a note on what that segment says about the question.

  3. Classify the notes

    Each margin is classified as relevant or irrelevant to the query, and only the useful notes are kept.

  4. Answer with notes in hand

    The relevant margins are added to the end of the prompt, so the final answer draws on both the full context and the distilled notes.

FIG. 03: Interactive figure

Read a long report the way the model does.

Prefill the document chunk by chunk, watch margin notes appear, then switch notes off and see how the final prompt changes.

Loading interactive demo…

FIG. 04: Notes

The problem

Models with long context windows still get “lost in the middle”. As the input grows, the relevant facts are drowned out by everything around them. Retrieval (RAG) helps, but it can miss what matters when the answer has to be pieced together across many parts of a document.

The approach

Writing in the Margins (WiM) uses the chunked prefill that long-context inference already performs. As each chunk enters the KV cache, the model writes a short margin note about that chunk. The notes are classified for relevance, and the useful ones are added to the end of the prompt before the final answer. The model reads the whole document and keeps notes as it goes, much as a careful human reader would.

The pattern adds little compute overhead and needs no fine-tuning. Because margins arrive while reading is still under way, they can also be shown to users as progress.

What we found

  • +7.5% average accuracy on reasoning tasks (HotpotQA, MultiHop-RAG).
  • More than 30% F1 improvement on aggregation tasks (CWE).
  • Works across off-the-shelf models, compared against both long-context and RAG baselines.

Reference

Cite this work
@misc{russak2024writing,
  title={Writing in the Margins: Better Inference Pattern for Long Context Retrieval},
  author={Melisa Russak and Umar Jamil and Christopher Bryant and Kiran Kamble and Axel Magnuson and Mateusz Russak and Waseem AlShikh},
  year={2024},
  eprint={2408.14906},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}