Why raw score blending is fragile

Dense retrieval ranks documents by similarity in an embedding space. Sparse retrieval with BM25 ranks documents using term occurrences and collection-level statistics. Both produce ordered result lists, but their numeric scores have different meanings and ranges.

A weighted formula such as dense_score + BM25_score is therefore difficult to tune safely. A change in embedding model, query shape, index configuration, or corpus can alter score distributions even when relevance behavior has not changed. Normalization can help in some systems, but it adds assumptions that should be tested against representative relevance judgments.

RRF avoids this score-calibration problem. It uses only each document’s position in each ranked list, preserving the useful signal that documents near the top of either retrieval method deserve attention.

  • Dense search can identify semantic paraphrases and conceptually related content.
  • BM25 can strongly reward exact terms, identifiers, product names, and rare vocabulary.
  • Rank positions are easier to combine than scores with incompatible scales.

Apply reciprocal rank fusion to two candidate lists

Run the same user query through dense search and BM25 search, then request a candidate depth large enough to create overlap and recover documents that one method may miss. For every unique document returned by either list, compute an RRF score from its rank in each list.

For a document d, the common formula is: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across available result lists. The constant k reduces the difference between adjacent top ranks and keeps one list from dominating solely because of a rank-one placement.

If a document is absent from a list, it contributes nothing for that list. Sort the union of candidates by the final RRF score, then return the top results or send that smaller set to a later reranking stage when your application has one.

  • Use 1-based ranks: the first result has rank 1.
  • Deduplicate by a stable document identifier before scoring.
  • Choose one k value as an initial policy and validate it with labeled queries.
  • Keep the original dense and BM25 ranks for debugging and relevance analysis.

Make fusion observable and testable

Hybrid retrieval is easier to improve when the response pipeline records why a document was selected. Store the dense rank, BM25 rank, fused score, and retrieval source for each candidate during evaluation. These fields reveal whether strong results are shared by both methods or rescued by only one of them.

Build a small query set from real application traffic, including exact identifiers, short ambiguous queries, natural-language questions, and domain-specific terminology. For each query, judge whether expected documents appear in the final top-k, not merely whether they appeared somewhere in either source list.

Operationally, define behavior for partial retrieval. If one source is unavailable or returns no candidates, RRF can still rank the remaining list. Treat that as a deliberate fallback policy, and surface the retrieval-source information so downstream teams can distinguish a hybrid response from a single-source response.

  • Evaluate final top-k relevance, candidate coverage, and failure cases separately.
  • Inspect queries where dense and BM25 disagree; they often expose vocabulary or embedding gaps.
  • Version retrieval settings alongside relevance evaluations.
  • Use stable document IDs so result merging remains deterministic.