Why rank fusion is safer than score fusion

Dense-search similarity values and BM25 scores are produced by different retrieval methods. Their numeric ranges, distributions, and meaning can vary with the query, corpus, and configuration. Adding those raw values together can make one retrieval path dominate for reasons unrelated to relevance.

RRF avoids that comparison. It uses each result’s position in its own ranked list, then rewards documents that appear near the top of either list. A document that performs well in both lists receives contributions from both rankings.

  • Use dense retrieval for semantic similarity and paraphrased questions.
  • Use BM25 for exact product names, error codes, IDs, and rare terms.
  • Fuse ranks rather than assuming dense and BM25 scores share a scale.

Apply RRF after retrieving both candidate lists

For a query, retrieve a bounded candidate list from the dense path backed by regional S3 Vectors and another from the Quickwit BM25 path. Join results by a stable document or chunk identifier. Then calculate an RRF score for every identifier seen in either list.

A common formula is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in retrieval list i. The constant k reduces the difference between adjacent top ranks; 60 is a commonly used starting value, not a universal requirement.

  • Keep rank numbering one-based: the first result has rank 1.
  • Use the same document or chunk ID across dense and sparse indexes.
  • Choose a candidate depth that leaves enough items for fusion and later filtering.
  • Treat a missing document from one list as contributing zero from that list.

Validate fusion with query-shaped evaluation sets

RRF is simple, but its effect should be evaluated against the queries your users actually ask. Build a small labeled set that includes semantic questions, exact-term lookups, mixed queries, and queries with acronyms or identifiers. Compare dense-only, BM25-only, and fused rankings using the same relevance judgments.

Review failures by query type rather than relying on one aggregate number. If exact identifiers are routinely missed, examine sparse retrieval and document tokenization. If paraphrases are weak, examine embedding and chunking choices. Fusion is most useful when each retrieval path contributes relevant items the other path does not.

  • Record the source rank and fused rank for each returned item.
  • Inspect duplicate or near-duplicate chunks before presenting final results.
  • Test several candidate depths and k values on held-out queries.
  • Keep retrieval fusion separate from any downstream reranking decision.