Why raw dense and BM25 scores should not be added blindly

A dense retrieval score and a BM25 score are produced by different models and different statistical assumptions. Even when each score is useful within its own result list, their numeric ranges and distributions may not be comparable across queries.

Adding uncalibrated scores can make one retriever dominate for accidental reasons, such as a wider score range rather than better relevance. Score normalization can help, but it requires careful validation and may behave differently as document collections or query patterns change.

  • Dense retrieval represents similarity in embedding space.
  • BM25 rewards overlap between query and document terms.
  • A score of 0.8 in one retrieval path does not inherently equal 0.8 in the other.
  • Rank positions are often safer to combine than raw scores.

Fuse two candidate lists with reciprocal rank fusion

Run dense search against the vector collection and BM25 search against the sparse index, then retain a candidate list from each path. RRF assigns each document a contribution based on its rank in every list where it appears. Documents that rank well in either list, and especially documents found by both, rise in the merged ranking.

For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across result lists. The constant k reduces the impact of tiny rank differences near the top and makes fusion less sensitive to one list placing a result first instead of third.

  • Choose a candidate depth for each retriever, such as the top N results.
  • Use one-based ranks: first result has rank 1.
  • Compute: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use a consistent document identifier to deduplicate candidates before returning results.

Make fusion observable and tune it with relevance judgments

Start with the same candidate depth for dense and sparse retrieval so that neither path is silently truncated. Then inspect merged results by query class: natural-language questions, product names, error messages, code-like identifiers, and short keyword searches often reveal different retrieval behavior.

RRF does not eliminate evaluation work. Maintain a small set of representative queries with expected relevant documents, compare dense-only, BM25-only, and fused rankings, and record failures. If a result is relevant but absent from both candidate lists, fusion cannot recover it; improve recall in the underlying retrieval path first.

  • Log which retrieval paths contributed to each final result.
  • Track candidate overlap between dense and sparse lists.
  • Evaluate ranking quality separately for semantic and exact-match query groups.
  • Keep fusion logic in the application layer when you need transparent iteration.