Retrieval systems compared
Comparative Analysis of Retrieval Systems in the Real World
- Identifier
- arXiv:2405.02048
- Published
- May 3, 2024
- Authors
- 2
- Type
- Paper
- Area
- Retrieval & knowledge
In one paragraph
Eight production retrieval setups compared on the RobustQA benchmark, measuring both accuracy on paraphrased questions and response time. A graph-based retrieval approach gave the best balance.
- RobustQA average score, graph search + LM (best of 8)
- 86.31
- RobustQA average score, graph search + LM (best of 8)
- its response time, tied for fastest
- <0.6s
- its response time, tied for fastest
- lowest score in the comparison (Amazon SageMaker RAG)
- 32.74
- lowest score in the comparison (Amazon SageMaker RAG)
- production retrieval setups compared
- 8
- production retrieval setups compared
FIG. 02: Mechanism
How it works
Choose a paraphrase benchmark
RobustQA asks questions in many paraphrased forms across domains such as finance, biomedicine and technology, so a system can't rely on matching the original wording.
Assemble eight pipelines
Production stacks are tested as built, including Azure Cognitive Search, Canopy, LangChain with Pinecone, LlamaIndex with Weaviate, Vertex AI Search, SageMaker and a graph-search approach.
Score accuracy
Each response is compared with gold answers across the paraphrased questions, giving a RobustQA average score per pipeline.
Time the response
Latency runs from the query request until a complete response reaches the user, adjusted for network effects. It is reported as a bound, such as under 0.6 seconds.
Compare both axes
Graph search with an LLM scored 86.31 in under 0.6 seconds. LlamaIndex with Weaviate came second at 75.89, and Amazon SageMaker RAG was lowest at 32.74.
FIG. 03: Interactive figure
Same question, eight ways, eight pipelines
Pick a pipeline, see which rephrasings it would still answer, and compare reported accuracy against response time.
Loading interactive demo…
FIG. 04: Notes
Summary
The paper benchmarks how real retrieval-augmented generation (RAG) stacks perform when users phrase the same question in different ways. It compares eight configurations on RobustQA accuracy and average response time:
- Azure Cognitive Search with GPT-4
- Pinecone Canopy
- LangChain with Pinecone and OpenAI
- LangChain with Pinecone and Cohere
- LlamaIndex with Weaviate hybrid search
- Google Vertex AI Search
- Amazon SageMaker RAG
- A graph-search approach paired with a language model
The graph-based approach gave the best balance of accuracy and speed. LlamaIndex with Weaviate also scored highly on accuracy.
Reference
@misc{mozolevskyi2024comparative,
title={Comparative Analysis of Retrieval Systems in the Real World},
author={Dmytro Mozolevskyi and Waseem AlShikh},
year={2024},
eprint={2405.02048},
archivePrefix={arXiv},
primaryClass={cs.IR}
}