Why raw dense and sparse scores should not be added

A dense-search score and a BM25 score represent different ranking signals. Their ranges, distributions, and sensitivity to query length can differ. Even if both values rise for better matches within their own result sets, a dense score of one value does not inherently mean the same thing as a BM25 score with the same—or a different—numeric value.

Adding uncalibrated scores can make one retriever dominate merely because its scoring scale is wider. Manual weights may appear to work on a small set of examples, then behave differently for short keyword queries, long natural-language questions, or queries containing product codes and names.

Rank-based fusion sidesteps this comparability problem. Instead of asking which score is larger, it asks where each document appeared in each ranked list.

  • Dense search is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact terms, rare tokens, and identifiers.
  • Raw scores from separate retrieval systems are not automatically interoperable.
  • A fusion method should preserve a document’s identity across both result lists.

Apply reciprocal rank fusion to two result lists

Run the dense query against the vector index and the sparse query against the BM25 index. Request a reasonably sized candidate list from each system, then merge candidates by a stable document identifier. For every document, add a contribution from each list in which it appears.

The common RRF formula is: RRF(d) = Σ 1 / (k + rank_i(d)). Here, rank_i(d) is the one-based rank of document d in result list i, and k is a positive constant that reduces the impact of small rank changes near the top of a list. A document returned by both retrievers receives two contributions.

After calculating the fused score, sort descending and return the top results. Keep the original dense and BM25 ranks in diagnostic data; they make it much easier to understand why a result was selected.

  • Use one-based ranks: the first result has rank 1.
  • Deduplicate documents before producing the final ranking.
  • Choose a stable ID shared by vector and sparse records.
  • Store per-retriever rank metadata for debugging and evaluation.

Tune candidate depth and evaluate query slices

RRF can only promote documents that appear in at least one candidate list. If each retriever returns too few candidates, useful documents may never reach fusion. Start with a candidate depth larger than the final number of results, then adjust it according to latency limits and offline relevance review.

Evaluate more than an overall average. Separate queries containing exact identifiers from broad conceptual questions, and include queries with synonyms, abbreviations, and mixed natural-language-plus-keyword phrasing. These slices reveal whether dense retrieval, BM25, or their combination is contributing useful coverage.

RRF is deliberately simple, but it is not a substitute for clean source documents, appropriate chunking, and a reliable relevance set. Treat fusion as one stage in a retrieval pipeline and inspect failures at the document, chunk, and query levels.

  • Fetch more candidates than the number of results shown to users.
  • Compare dense-only, BM25-only, and fused rankings on the same judged queries.
  • Review failures by query type instead of relying only on aggregate metrics.
  • Revisit chunking and metadata filters when relevant content is absent from both lists.