Why raw-score mixing is fragile

A dense retriever returns a similarity-derived score, while BM25 returns a lexical relevance score based on term frequency and document statistics. Even when both scores are useful within their own result lists, their magnitudes, distributions, and sensitivity to query wording differ.

Adding or averaging raw scores assumes that a given numerical value means the same thing in both systems. That assumption often fails across query types, document collections, embedding models, and BM25 configurations. A rank-based fusion step removes the need to establish a universal score scale.

  • Dense retrieval can surface semantically related text when query wording differs from document wording.
  • BM25 can strongly reward exact identifiers, uncommon terms, quoted phrases, and product names.
  • Score ranges can shift as the corpus, embedding model, or sparse index settings change.

Fuse the two lists with RRF

Run dense search and BM25 independently for the same query, then retain a candidate list from each. For every document, assign an RRF contribution based on its position in each list: 1 divided by k plus the document rank. Sum contributions when a document appears in both lists.

The formula is RRF(d) = Σ 1 / (k + rankᵢ(d)), where rankᵢ(d) is the one-based rank of document d in result list i. The constant k softens the difference between nearby ranks, preventing the first few positions from dominating too aggressively.

  • Use one-based ranks: the first result has rank 1.
  • Assign no contribution from a retriever when the document is absent from its candidate list.
  • Deduplicate by a stable document or chunk identifier before sorting fused results.
  • Start with the same candidate depth for dense and sparse retrieval, then tune using representative queries.

Make fusion observable and easy to tune

Store the dense rank, BM25 rank, and fused score alongside each returned result during evaluation. This makes it possible to see whether a result won because both retrievers agreed, because BM25 matched an exact term, or because dense retrieval found a semantic neighbor.

Evaluate changes with a fixed query set that includes exact-name searches, natural-language questions, abbreviations, and terminology that is likely to be absent from an embedding model's training distribution. Adjust candidate depth and k deliberately, and compare relevance judgments rather than relying on fused-score magnitude.

  • Inspect queries where dense and sparse top results have little overlap.
  • Track whether relevant documents are present in either candidate list before judging the fusion rule.
  • Keep retriever-specific ranks in logs to diagnose ranking regressions.
  • Re-evaluate after changing corpus content, chunking, embedding generation, or BM25 analysis.