Why raw dense and sparse scores should not be added

A dense-search score and a BM25 score are produced by different ranking models. Their ranges, distributions, and meanings depend on index configuration, query text, document length, and implementation details. A score of 0.7 from one retriever is not inherently equivalent to a score of 0.7, or 7, from another.

Adding or averaging raw scores can therefore make ranking sensitive to accidental scale differences rather than relevance. It may work for a narrow test set, then change behavior when a new corpus, embedding model, or query mix shifts the score distribution. Score normalization can be useful, but it requires measurement and ongoing validation.

  • Dense retrieval favors semantic proximity in embedding space.
  • BM25 favors lexical evidence, including exact terms and uncommon tokens.
  • Score scales are retriever-specific, not a shared relevance unit.
  • A fusion method based on rank avoids a premature calibration problem.

Fuse two candidate lists with reciprocal rank fusion

Run dense retrieval and BM25 retrieval for the same query, then retain a candidate list from each. For every document, RRF adds a contribution based on its position in each list: RRF(d) = Σ 1 / (k + rankᵢ(d)). The sum is taken over the lists in which document d appears.

The constant k reduces the gap between adjacent top ranks and prevents a single first-place result from overwhelming all other evidence. A commonly used starting value is 60, but it is a tuning choice rather than a universal rule. Documents returned by both retrievers receive two contributions, while documents found by only one route can still be retained.

  • Fetch a fixed top-N candidate list from dense retrieval.
  • Fetch a fixed top-N candidate list from BM25 retrieval.
  • Assign ranks starting at 1 within each list.
  • Sum RRF contributions, sort descending, and return the top results.

Make fusion observable before making it complex

Start with a small evaluation set that reflects real searches: natural-language questions, short keyword queries, product codes, quoted phrases, and queries with mixed terminology. For each query, record whether relevant documents came from dense retrieval, sparse retrieval, or both. This reveals the complementarity that hybrid retrieval is meant to capture.

Operational logging should preserve more than the final fused order. Keep the per-retriever rank, whether a result was present in each candidate list, the fusion parameters, and the document identifier. When a result looks surprising, these fields make it possible to determine whether the issue was candidate generation or fusion.

  • Tune candidate-list depth separately from the final number of results shown.
  • Compare dense-only, BM25-only, and RRF-fused rankings on the same query set.
  • Review failures involving exact identifiers and terminology changes.
  • Change one parameter at a time and keep evaluation queries versioned.