Fusion-in-Decoder QA

Achieving State-of-the-Art Open-Domain QA Performance through Fusion-in-Decoder

FIG. 01 Retrieve passages → Encode each passage separately → Fuse in the decoder → Scale to many passages
Identifier
WRITER Research
Published
Sep 13, 2023
Authors
WRITER
Type
Paper
Area
Retrieval & knowledge

In one paragraph

Open-domain question answering with a Fusion-in-Decoder architecture. Retrieved passages are encoded independently and fused in the decoder, so the model can reason across many sources at once.

FIG. 02: Mechanism

How it works

  1. Retrieve passages

    A retriever such as BM25 or a dense retriever returns the top passages for the question, typically as 100-word chunks.

  2. Encode each passage separately

    Each passage is paired with the question and its title and run through the encoder on its own, so self-attention never spans two passages.

  3. Fuse in the decoder

    The encoded passages are concatenated, and the decoder cross-attends over all of them while it writes, so it can combine evidence that no single passage contains.

  4. Scale to many passages

    Encoder cost grows linearly with the number of passages rather than quadratically. In the original FiD paper (Izacard & Grave, 2021), accuracy kept improving up to 100 passages.

FIG. 03: Interactive figure

Fuse evidence from six passages

Switch between one long encoder input and Fusion-in-Decoder, pick the part of the answer being written, then drag the passage count to compare encoder cost.

Loading interactive demo…

FIG. 04: Notes

Summary

In Fusion-in-Decoder (FiD), each retrieved passage is encoded separately together with the question, and the decoder attends over all of the encoded passages at once to generate the answer. Evidence spread across many documents can then be combined without a single, very long encoder input. This work applies the approach to open-domain question answering.

Reference