Why raw score blending is fragile

A dense retriever typically ranks documents by vector similarity, while BM25 ranks them from term-frequency and corpus-statistics signals. Even when both return a numeric score, the magnitude and distribution of those values are not inherently equivalent.

Adding or averaging raw scores can therefore make one retriever dominate for reasons unrelated to relevance. The problem becomes more pronounced when document collections, query types, analyzers, or embedding models change over time.

  • Dense retrieval is useful for semantic matches and paraphrases.
  • BM25 is useful when exact terms, identifiers, and rare vocabulary matter.
  • A score of 1 from one retrieval method does not have a universal meaning relative to a score of 1 from another.
  • Rank-based fusion avoids requiring a shared score calibration.

Fuse two ranked lists with RRF

Run the same query through dense search and through Quickwit BM25, then retain a bounded candidate list from each result. For every unique document, calculate an RRF value by adding 1 divided by k plus its rank from each list. Documents absent from a list contribute nothing from that list.

The constant k reduces the difference between adjacent top positions and prevents the first result from overwhelming the merged ranking. A commonly used starting value is 60, but it is a tuning parameter rather than a universal rule.

  • Use one-based ranks: the first result has rank 1.
  • Formula: RRF(document) = Σ 1 / (k + rank).
  • Deduplicate by a stable document or chunk identifier before sorting.
  • Sort by descending fused value, then apply a deterministic tie-breaker such as document ID.

Make fusion operationally predictable

Keep the candidate depth intentionally limited. If an application needs a final top 10, requesting a moderate pool from each retriever gives RRF room to surface documents supported by either method without turning every query into an excessively broad merge.

Evaluate fused results with a representative query set that includes semantic questions, exact-name lookups, abbreviations, and queries containing both concepts and identifiers. Review not only aggregate relevance judgments but also failure cases where an important exact match or paraphrase disappears from the final list.

  • Start with the same candidate depth for dense and BM25, then adjust based on observed query behavior.
  • Log each document's dense rank, BM25 rank, and final fused rank for debugging.
  • Keep retrieval configuration versioned so ranking changes can be traced to a model, analyzer, or fusion update.
  • Use RRF as a first-stage merge; any later reranking should operate on the fused candidate set.