Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different retrieval models with different scoring behavior. Their numeric ranges, distributions, and meaning can vary across indexes, queries, and implementation details. Adding them together without calibration can cause one retriever to dominate merely because its scores use a larger scale.

Rank-based fusion avoids that problem. Instead of treating a score as a universal measure of relevance, it uses each system’s ordering of documents. This is useful when you want a stable first hybrid baseline before investing in query-specific weighting or learned reranking.

  • Dense retrieval helps with paraphrases, semantic similarity, and related concepts.
  • BM25 helps preserve exact matches for tokens such as SKUs, acronyms, log messages, and version strings.
  • Rank fusion combines evidence without requiring score normalization.

Apply reciprocal rank fusion to the two result lists

For a query, retrieve a candidate list from the dense index and another from the BM25 index. Then assign every document a fused score based on its rank in each list: RRF(d) = Σ 1 / (k + rank_i(d)). The sum includes each retrieval list in which document d appears.

The constant k reduces the gap between adjacent top ranks and keeps a single first-place result from overwhelming the rest of the combined evidence. A common starting value is 60, but it is a tuning parameter rather than a universal rule. Keep document identifiers consistent across the dense and sparse indexes so duplicate candidates can be merged correctly.

  • Request the same candidate depth from both retrievers as an initial baseline.
  • Use one canonical document or chunk ID across S3 Vectors and Quickwit.
  • Assign no contribution for a document missing from a given result list.
  • Sort by fused score, then pass only the top fused candidates to subsequent context selection or reranking.

Evaluate by query type, not only by a single aggregate metric

Hybrid retrieval is most useful when the query set contains multiple retrieval modes. Build a small labeled evaluation set that includes conceptual questions, exact-identifier lookups, product terminology, short ambiguous queries, and queries with alternate wording. Compare dense-only, BM25-only, and fused rankings against the same relevance judgments.

Inspect failures in addition to summary metrics. If identifier queries disappear, the sparse candidate depth may be too shallow. If semantic queries are crowded out by repeated keywords, inspect chunking and the balance of candidate lists. RRF is intentionally simple: its value is providing an understandable baseline that exposes where retrieval needs more targeted work.

  • Track whether at least one relevant chunk appears near the top of each ranking.
  • Segment results by query class, especially exact-match versus semantic queries.
  • Log source rank and fused rank for retrieved chunks to make debugging concrete.
  • Change one variable at a time: candidate depth, k, chunking strategy, or post-retrieval reranking.