Why raw dense and BM25 scores should not be added blindly

A dense retriever ranks documents by similarity in an embedding space. A BM25 retriever ranks documents using term occurrence and document-level statistics. Although both produce scores, those scores come from different models and scales.

Adding raw scores can make one retriever dominate simply because its numeric range is wider, not because its result is more useful. Score normalization can help in some systems, but it introduces choices about distributions, query behavior, and calibration. Rank-based fusion avoids requiring those scores to mean the same thing.

  • Dense retrieval can surface semantically related wording.
  • BM25 can strongly reward exact terms, identifiers, and uncommon vocabulary.
  • A document's position in each ranked list is comparable even when the underlying scores are not.

Apply Reciprocal Rank Fusion to two result lists

Run the same user query through both retrieval paths, then keep a bounded candidate list from each. For every document returned by either path, calculate an RRF score using its rank in each list: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across the lists where document d appears.

The constant k reduces the difference between nearby ranks and keeps the highest positions from overwhelming the rest of the fused list. It is a tuning parameter rather than a universal truth: choose a value, inspect representative queries, and adjust only when the observed ranking behavior supports it.

  • Treat rank as 1-based: the first result has rank 1.
  • Assign no contribution for a source where the document was not returned.
  • Deduplicate by a stable document identifier before sorting the fused results.
  • Sort descending by fused score and return the requested top results.

Make fusion observable and safe to tune

Fusion is easier to improve when each result carries retrieval provenance. Record whether a document came from dense search, BM25, or both, along with its per-source ranks and final fused score. This makes it possible to investigate why a result appeared without treating the fused order as a black box.

Use a fixed evaluation set that reflects real query patterns: natural-language questions, exact product names, error strings, abbreviations, and mixed queries. Compare ranking changes qualitatively or against available relevance judgments. Change one variable at a time, such as candidate depth or k, so the effect of each adjustment remains interpretable.

  • Log source membership, source rank, and final rank for each returned document.
  • Keep dense and sparse candidate depths explicit in configuration.
  • Test empty or low-result responses from either retrieval path.
  • Preserve source-specific metadata for debugging and downstream presentation.