Why raw dense and BM25 scores should not be added directly

A dense-search score and a BM25 score are produced by different ranking models. Their ranges, distributions, and meanings can vary with embedding choice, similarity metric, corpus composition, analyzer settings, and query length. A score of 0.7 from one dense retrieval setup is not inherently comparable to a BM25 score of 7.

Adding raw scores can therefore create accidental weighting. A retrieval path may dominate simply because its scoring scale is numerically larger, not because it produced better results. Normalization can help, but it requires choices and ongoing validation as indexes, embeddings, and query traffic change.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact vocabulary, codes, names, and uncommon terms.
  • Score scales are implementation-dependent.
  • Rank positions are often easier to combine than raw scores.

Apply reciprocal rank fusion to two candidate lists

RRF combines ranked lists rather than their raw scores. Retrieve a candidate set from dense search and another from BM25, then assign each document a fusion contribution based on its position in each list. Documents returned by both paths accumulate contributions; documents high in either list can still remain competitive.

For each document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across retrieval lists in which d appears. The rank is one-based, and k is a positive constant that reduces the difference between adjacent top positions. A commonly used starting value is 60, but it should be treated as a tunable application choice rather than a universal optimum.

  • Request the same candidate depth from dense and sparse retrieval when practical.
  • Deduplicate by a stable document or chunk identifier before ranking.
  • Assign rank 1 to the first result in each list.
  • Sort documents by descending fused score and retain the top results.

Make fusion observable before making it more complex

Log the source ranks that produced each fused result: dense rank, BM25 rank, fused score, and the query category if available. These fields make it possible to identify patterns such as identifier-heavy queries being carried by BM25 or conceptual questions being carried by dense retrieval.

Evaluate with representative queries and relevance judgments whenever possible. Compare dense-only, BM25-only, and RRF results at the same cutoff, such as the top 5 or top 10. If important query classes need different behavior, add routing or weighting only after the baseline reveals a repeatable reason to do so.

  • Keep a small test set containing semantic, exact-match, and mixed queries.
  • Inspect failures where neither retrieval path returns a relevant candidate.
  • Check that chunks from the same source do not overwhelm the final list.
  • Re-evaluate after changing embeddings, chunking, analyzers, or corpus content.