Retrieval systems compared

Comparative Analysis of Retrieval Systems in the Real World

Dmytro Mozolevskyi, Waseem AlShikh

FIG. 01 Choose a paraphrase benchmark → Assemble eight pipelines → Score accuracy → Time the response → Compare both axes
Identifier
arXiv:2405.02048
Published
May 3, 2024
Authors
2
Type
Paper
Area
Retrieval & knowledge

In one paragraph

Eight production retrieval setups compared on the RobustQA benchmark, measuring both accuracy on paraphrased questions and response time. A graph-based retrieval approach gave the best balance.

RobustQA average score, graph search + LM (best of 8)
86.31
RobustQA average score, graph search + LM (best of 8)
its response time, tied for fastest
<0.6s
its response time, tied for fastest
lowest score in the comparison (Amazon SageMaker RAG)
32.74
lowest score in the comparison (Amazon SageMaker RAG)
production retrieval setups compared
8
production retrieval setups compared

FIG. 02: Mechanism

How it works

  1. Choose a paraphrase benchmark

    RobustQA asks questions in many paraphrased forms across domains such as finance, biomedicine and technology, so a system can't rely on matching the original wording.

  2. Assemble eight pipelines

    Production stacks are tested as built, including Azure Cognitive Search, Canopy, LangChain with Pinecone, LlamaIndex with Weaviate, Vertex AI Search, SageMaker and a graph-search approach.

  3. Score accuracy

    Each response is compared with gold answers across the paraphrased questions, giving a RobustQA average score per pipeline.

  4. Time the response

    Latency runs from the query request until a complete response reaches the user, adjusted for network effects. It is reported as a bound, such as under 0.6 seconds.

  5. Compare both axes

    Graph search with an LLM scored 86.31 in under 0.6 seconds. LlamaIndex with Weaviate came second at 75.89, and Amazon SageMaker RAG was lowest at 32.74.

FIG. 03: Interactive figure

Same question, eight ways, eight pipelines

Pick a pipeline, see which rephrasings it would still answer, and compare reported accuracy against response time.

Loading interactive demo…

FIG. 04: Notes

Summary

The paper benchmarks how real retrieval-augmented generation (RAG) stacks perform when users phrase the same question in different ways. It compares eight configurations on RobustQA accuracy and average response time:

  • Azure Cognitive Search with GPT-4
  • Pinecone Canopy
  • LangChain with Pinecone and OpenAI
  • LangChain with Pinecone and Cohere
  • LlamaIndex with Weaviate hybrid search
  • Google Vertex AI Search
  • Amazon SageMaker RAG
  • A graph-search approach paired with a language model

The graph-based approach gave the best balance of accuracy and speed. LlamaIndex with Weaviate also scored highly on accuracy.

Reference

Cite this work
@misc{mozolevskyi2024comparative,
  title={Comparative Analysis of Retrieval Systems in the Real World},
  author={Dmytro Mozolevskyi and Waseem AlShikh},
  year={2024},
  eprint={2405.02048},
  archivePrefix={arXiv},
  primaryClass={cs.IR}
}