Why raw dense and sparse scores should not be added

A dense-search score and a BM25 score are not automatically measurements on the same scale. Their values depend on the similarity metric, index configuration, query terms, document length effects, and implementation details. A document with a high score in one system is not necessarily more relevant than a document with a numerically lower score in the other.

Adding scores directly can therefore make one retrieval path dominate merely because its values have a wider range. Min-max normalization can appear to solve this, but it is sensitive to the particular candidates returned for each query. Rank-based fusion avoids treating the score values as interchangeable.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful when exact terminology, identifiers, and rare terms matter.
  • Score magnitudes from separate retrieval systems should be treated as incomparable by default.

Fuse top-k lists with Reciprocal Rank Fusion

Run the same user query through the dense and sparse retrieval paths, requesting a bounded candidate list from each. Then combine documents by a stable canonical document ID. For every occurrence of a document at rank r in a result list, add 1 divided by k plus r to its fused score.

The RRF formula is: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is over each result list containing document d. The constant k reduces the difference between nearby ranks and prevents the first position from overwhelming all other evidence. A commonly used starting point is k = 60, but it should be treated as a tunable parameter rather than a universal optimum.

  • Retrieve, for example, the top 50 to 100 candidates from each path.
  • Use one canonical ID so the same document is merged rather than shown twice.
  • Assign ranks starting at 1 and omit lists where the document was not returned.
  • Sort by fused score, then apply a deterministic tie-breaker such as document ID.

Make fusion observable and safe to tune

Store retrieval metadata with each fused result: whether it came from dense search, BM25, or both; its rank in each list; and its final RRF score. This makes it possible to investigate surprising results without needing to infer behavior from a single opaque number.

Evaluate changes against representative queries before changing candidate depth or the RRF constant. Include queries with product names, error codes, acronyms, natural-language questions, and mixed terminology. The goal is not simply to maximize overlap between lists; it is to preserve useful exact matches while recovering semantically relevant documents that keyword retrieval may miss.

  • Log source ranks and candidate-list sizes for every fused response.
  • Keep a small judged query set for regression checks.
  • Test no-result and one-source-only cases explicitly.
  • Version fusion settings alongside other retrieval configuration.