Fusion-in-Decoder QA
Achieving State-of-the-Art Open-Domain QA Performance through Fusion-in-Decoder
- Identifier
- WRITER Research
- Published
- Sep 13, 2023
- Authors
- WRITER
- Type
- Paper
- Area
- Retrieval & knowledge
In one paragraph
Open-domain question answering with a Fusion-in-Decoder architecture. Retrieved passages are encoded independently and fused in the decoder, so the model can reason across many sources at once.
FIG. 02: Mechanism
How it works
Retrieve passages
A retriever such as BM25 or a dense retriever returns the top passages for the question, typically as 100-word chunks.
Encode each passage separately
Each passage is paired with the question and its title and run through the encoder on its own, so self-attention never spans two passages.
Fuse in the decoder
The encoded passages are concatenated, and the decoder cross-attends over all of them while it writes, so it can combine evidence that no single passage contains.
Scale to many passages
Encoder cost grows linearly with the number of passages rather than quadratically. In the original FiD paper (Izacard & Grave, 2021), accuracy kept improving up to 100 passages.
FIG. 03: Interactive figure
Fuse evidence from six passages
Switch between one long encoder input and Fusion-in-Decoder, pick the part of the answer being written, then drag the passage count to compare encoder cost.
Loading interactive demo…
FIG. 04: Notes
Summary
In Fusion-in-Decoder (FiD), each retrieved passage is encoded separately together with the question, and the decoder attends over all of the encoded passages at once to generate the answer. Evidence spread across many documents can then be combined without a single, very long encoder input. This work applies the approach to open-domain question answering.
Reference