Why raw score blending is fragile

A dense retrieval score and a BM25 score do not necessarily share the same range, distribution, or interpretation. A score of 0.7 from one retrieval system is not inherently comparable to a score of 0.7 from another.

Normalizing scores can help in controlled situations, but it introduces choices about score distributions, query types, and changing indexes. Rank-based fusion avoids treating unlike scores as if they were directly equivalent.

  • Dense retrieval favors semantic similarity.
  • BM25 favors lexical overlap and term specificity.
  • Score ranges can shift as content, embeddings, or index settings change.
  • Rank positions are easier to compare across retrieval methods.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its position in each result list. For a document d, calculate: RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i is the one-based rank in retrieval list i and k is a positive constant.

Retrieve a bounded candidate set from dense search and another from BM25, deduplicate by document identifier, add each document’s contributions, and sort by the resulting fused score. Documents that rank well in either list can appear near the top; documents supported by both lists gain additional weight.

  • Use stable, shared document IDs for deduplication.
  • Treat rank 1 as the first result in each list.
  • Choose one k value and evaluate it against representative queries.
  • Keep the original dense and BM25 ranks for debugging.

Make fusion observable before tuning it

Start with a small set of real queries that includes semantic paraphrases, exact technical strings, product names, and mixed queries. For each query, inspect the dense list, the BM25 list, and the fused list rather than judging only the final order.

Logging source ranks makes failures actionable. If exact identifiers disappear, inspect sparse recall and candidate depth. If paraphrased questions miss relevant content, inspect embedding coverage and dense candidate depth. Fusion should combine useful candidates, not conceal weak retrieval inputs.

  • Record query text, document ID, source ranks, and fused rank.
  • Review queries where dense and sparse retrieval strongly disagree.
  • Use relevance judgments from your domain when available.
  • Tune candidate counts and k only after establishing a review set.