Why raw score blending is fragile

A dense retrieval score and a BM25 score are not naturally interchangeable. Their ranges, distributions, and meanings depend on the underlying retrieval method, index configuration, query wording, and corpus. Adding them directly can make one retriever dominate simply because its scores happen to occupy larger numeric values.

Score normalization can help, but it introduces choices that must be monitored over time. Min-max normalization is sensitive to outliers, while distribution-based approaches require representative query data. RRF avoids this first problem by using result positions rather than the score values produced by either retriever.

  • Dense retrieval contributes semantic matches and paraphrases.
  • BM25 contributes exact terminology, product names, codes, and uncommon tokens.
  • Rank positions are easier to combine than unrelated score scales.

Apply reciprocal rank fusion to two result lists

Run the same user query through the dense and sparse retrieval paths, requesting a sufficiently deep candidate list from each. For every document returned by either path, add a contribution based on its rank: 1 divided by k plus the document rank. The final RRF score is the sum of those contributions across lists.

The constant k reduces the impact of small rank changes near the top of a list. A commonly used starting value is 60, but it is a tuning parameter rather than a universal rule. Evaluate it against representative queries, especially queries containing both natural-language intent and exact terms.

  • Collect the top N document IDs from regional S3 Vectors and Quickwit BM25.
  • Assign rank 1 to the first result in each individual list.
  • Compute RRF(d) = Σ 1 / (k + rank_i(d)).
  • Sort unique documents by fused score and return the top results.

Make fusion observable before making it complex

Log the dense rank, sparse rank, and fused rank for each returned document. These fields make it possible to diagnose whether one retrieval path is consistently carrying useful results, whether duplicate documents are being merged correctly, and whether a query class needs different candidate depths.

Start with equal treatment of the two lists. If evaluation shows a consistent reason to prefer one path, use weighted RRF by multiplying each list’s contribution by a chosen weight. Keep the change explicit and test it on held-out queries; weights can improve one query type while weakening another.

  • Deduplicate by a stable document or chunk identifier before final sorting.
  • Record which retriever contributed each candidate.
  • Review failures for exact-match queries, paraphrase queries, and mixed queries.
  • Re-evaluate fusion settings when content, chunking, or embeddings change.