Why merge ranks instead of raw scores?

Dense-search similarity values and BM25 scores are not inherently comparable. Their ranges, distributions, and sensitivity to query length can differ, even when both searches are useful for the same corpus. Adding the two raw scores without calibration can unintentionally let one retrieval method dominate.

RRF avoids requiring a shared score scale. It uses each document's position in a ranked list rather than its original score. This makes it a useful baseline when combining a dense result set from S3 Vectors with a sparse BM25 result set from Quickwit.

  • Dense retrieval helps with paraphrases and semantic similarity.
  • BM25 helps with exact terms, identifiers, and rare vocabulary.
  • Rank-based fusion reduces dependence on incompatible score ranges.

Apply reciprocal rank fusion at the application layer

Request the top N candidates from dense retrieval and the top N candidates from BM25 retrieval for the same query. Create a map keyed by stable document or chunk ID. For every occurrence of an item at rank r, add 1 divided by k plus r to its fusion score. The constant k reduces the difference between adjacent top ranks; a commonly used starting value is 60, but it should be treated as a tuning parameter rather than a universal rule.

After accumulating scores from both lists, sort items by the fused score and return the top results. If a document appears in both lists, it receives contributions from both ranks. If it appears in only one list, it can still be returned when it ranks strongly in that retrieval channel.

  • Use the same chunk IDs in dense and sparse indexes.
  • Retrieve enough candidates from each channel before fusing.
  • Deduplicate by chunk ID before presenting results.
  • Keep original dense and BM25 ranks in logs for debugging.

Evaluate fusion with query categories, not anecdotes

A hybrid strategy should be evaluated against representative queries and expected relevant documents. Include semantic questions, exact-name searches, abbreviated terms, typo-prone queries, and queries containing codes or numbers. A single impressive example is not enough to establish that fusion improves retrieval for a production workload.

Review failures by retrieval channel. If BM25 finds an exact identifier that dense retrieval misses, that supports retaining sparse retrieval. If dense retrieval finds a relevant passage despite different terminology, that supports retaining semantic retrieval. These observations can guide candidate depth, query preprocessing, and later reranking experiments.

  • Build a small labeled query set from real search intent.
  • Measure whether relevant items appear near the top of the fused list.
  • Compare dense-only, BM25-only, and fused rankings.
  • Inspect queries where fusion lowers a previously relevant result.