Why raw score blending is fragile

A dense-search score and a BM25 score are produced by different ranking models and usually have different ranges, distributions, and meanings. Adding them together assumes that a one-point change in one system is equivalent to a one-point change in the other. That assumption is rarely stable across collections or query types.

Score normalization can help in some controlled settings, but it introduces more choices: which normalization method to use, what result window to sample, and how to handle queries with unusually flat or unusually sharp score distributions. RRF avoids this calibration problem by using rank position rather than the original score.

  • Dense retrieval helps with paraphrases and conceptual similarity.
  • BM25 preserves strong matches for exact words, codes, and uncommon terms.
  • Rank positions are easier to compare across retrieval methods than raw scores.

Apply reciprocal rank fusion to two result lists

Retrieve a candidate list from dense search and another from BM25 using the same query. For every document that appears in either list, add a contribution based on its rank in each list: 1 divided by k plus the rank. Documents present in both lists accumulate two contributions, while documents that rank highly in one list can still remain competitive.

The constant k reduces the difference between adjacent ranks near the top of a list. A commonly used starting value is 60, but it should be treated as a tuning parameter rather than a universal default. Keep the per-retriever candidate depth large enough that useful overlap and complementary results can reach the fusion stage.

  • Use one-based ranks: rank 1 is the first result.
  • Compute: fused_score(document) = Σ 1 / (k + rank).
  • Deduplicate by a stable document or chunk identifier before returning results.
  • Sort documents by fused score, then apply any downstream reranking or filtering.

Tune with query slices, not a single average

Evaluate fusion on query groups that reflect production traffic. Include exact-title queries, identifier-heavy queries, short natural-language questions, and longer descriptive requests. A fusion policy that improves semantic questions but pushes down exact matches may be inappropriate for a corpus where users frequently search for product names, error codes, or legal clauses.

Start with equal treatment for the dense and BM25 lists, then inspect failures before adding complexity. If one retrieval method is consistently weak for a defined query class, address the cause first: chunking, metadata filtering, analyzer configuration, query construction, or embedding choice. Fusion is most useful when each retriever contributes genuinely different relevant candidates.

  • Record the source ranks that contributed to each fused result.
  • Compare dense-only, BM25-only, and fused result sets on the same judged queries.
  • Check whether relevant documents appear in both lists or only one.
  • Keep a small set of known exact-match queries as regression tests.