Why raw-score blending is fragile

Dense retrieval scores describe the relationship between a query embedding and document embeddings. BM25 scores are produced from lexical term statistics, including factors such as term frequency and document length. Even when both systems rank relevant documents highly, their numeric ranges and distributions need not align.

A weighted formula such as dense_score + sparse_score can therefore behave unpredictably. A tuning value that works for one corpus, query shape, or indexing configuration may over-favor one retriever after content changes. Score normalization can help in some systems, but it introduces assumptions that must be monitored and retuned.

RRF avoids this calibration problem by using each document’s position in a ranked list rather than its original score.

  • Dense search can surface semantically related content when wording differs.
  • BM25 can preserve exact terms, identifiers, error messages, and uncommon names.
  • Raw dense and BM25 scores should not be assumed to share a common scale.

Fuse two result lists with RRF

Run the same user query through a dense retrieval path and a BM25 retrieval path. Request a candidate list from each, then deduplicate documents by a stable document identifier. For every occurrence of a document at rank r, add 1 / (k + r) to its fused score.

The constant k reduces the difference between adjacent top ranks and prevents a single first-place result from overwhelming all other evidence. A commonly used starting point is 60, but it is a parameter to evaluate against representative queries rather than a universal optimum.

After summing contributions from the dense and sparse lists, sort documents by the fused score. Documents found by both retrieval methods tend to rise, while a strong result from either method can still remain in the candidate set.

  • Use one-based ranks: the first result has r = 1.
  • Apply the same k initially to both retrievers for a clear baseline.
  • Fetch enough candidates from each path to allow useful overlap and complementary results.
  • Keep the original dense and BM25 ranks for debugging and evaluation.

Make fusion observable before making it complex

Record which retrieval path contributed each fused result, along with its rank in that path. This makes it possible to distinguish results supported by both systems from results supported only by dense or sparse retrieval. It also helps identify query classes where one path consistently provides the useful candidates.

Evaluate RRF with a small, representative query set containing semantic paraphrases, exact identifiers, multi-term questions, and domain vocabulary. Review not only whether a relevant document appears, but whether it is positioned high enough for the downstream experience.

If retrieval needs evolve, RRF remains a useful baseline. More elaborate methods—such as query-dependent weighting or reranking—should be compared against this baseline with recorded judgments rather than introduced solely because they are more complex.

  • Log fused rank, source ranks, and document identifiers.
  • Include both natural-language and exact-match queries in evaluation.
  • Inspect misses caused by candidate depth before changing fusion logic.
  • Treat RRF as a transparent baseline for future retrieval experiments.