Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different retrieval models with different score distributions. Even when both systems rank relevant documents well, a score of 0.8 from one system does not inherently mean the same thing as a score of 0.8 from the other.

Adding or averaging uncalibrated scores can make one retriever dominate because of its numeric range rather than because it found better evidence. Rank-based fusion avoids that assumption: it uses each system’s ordering while ignoring the incomparable magnitude of its scores.

  • Dense retrieval can help when query and document use different wording.
  • BM25 can preserve exact matches for product names, codes, and rare terms.
  • Raw-score fusion requires calibration that may vary by corpus, query type, and model.
  • Rank fusion provides a simple baseline before introducing learned rerankers.

Fuse two candidate lists with RRF

For a query, request a ranked dense list from S3 Vectors and a ranked sparse list from Quickwit BM25. For every document appearing in either list, assign an RRF contribution from each list where it appears: 1 divided by k plus that document’s one-based rank. Sum the contributions, then sort documents by the total.

The constant k reduces the advantage of being ranked at the very top and lets documents supported by both retrievers rise naturally. A commonly used starting value is 60, but it is a configuration choice rather than a universal optimum. Keep the implementation explicit so it can be evaluated on representative queries.

  • Use stable document IDs shared by the dense and sparse indexes.
  • Retrieve a sufficiently deep candidate list from each source before fusion.
  • Treat a missing document in one list as a zero contribution from that list.
  • Deduplicate by document ID before returning fused results.

Make fusion observable and easy to tune

Store the fused score alongside diagnostic fields such as dense rank, BM25 rank, and which retrievers returned the document. These fields make it possible to explain why a result appeared and to identify queries where one retrieval path consistently contributes little.

Evaluate changes with a fixed query set that reflects real retrieval tasks: semantic questions, exact identifier lookups, short keyword searches, and mixed queries. Compare ranked outputs and relevance judgments before changing candidate depths, the RRF constant, or adding a later reranking stage.

  • Log per-source ranks, not only the final fused rank.
  • Inspect queries where the top results come from only one retriever.
  • Version embedding models, index contents, and fusion settings together.
  • Keep a fallback behavior for cases where one retrieval source is unavailable.