Why score-based merging is often fragile

Dense retrieval and BM25 do not naturally produce scores with the same meaning. A dense-search score depends on the embedding model, vector normalization, and similarity metric. A BM25 score depends on term frequency, document length, corpus statistics, and query terms. Adding or averaging those values can make one retrieval path dominate for reasons unrelated to relevance.

Score normalization can help in controlled settings, but it adds operational assumptions. Score distributions may change when documents are added, an embedding model changes, or a query has unusually specific vocabulary. Rank-based fusion avoids requiring dense and sparse scores to share a common scale.

  • Treat dense and sparse scores as method-specific signals.
  • Request a ranked candidate list from each retrieval path.
  • Keep the original per-method scores for debugging, even if fusion uses ranks.
  • Evaluate fusion on representative queries rather than relying on score intuition.

Fuse ranked lists with Reciprocal Rank Fusion

RRF assigns a contribution to each document based on its rank in every candidate list where it appears. For a document d, the fused score is the sum of 1 divided by k plus the rank of d in each list. The constant k reduces the advantage of a single top position and makes the method less sensitive to small rank changes.

For example, if a document ranks second in the dense list and fifth in the BM25 list, it receives contributions from both positions. A document found by only one method can still rank well, but a document consistently surfaced by both methods gains evidence from agreement. This is useful when semantic similarity and lexical matching each catch different relevant results.

  • Retrieve the top N dense candidates from S3 Vectors.
  • Retrieve the top N sparse candidates from Quickwit BM25.
  • Deduplicate documents by a stable document or chunk identifier.
  • Compute fused_score(d) = Σ 1 / (k + rank_i(d)) and sort descending.

Choose candidate depth and inspect failures

The candidate depth N is a recall decision. If N is too small, a relevant result that appears just below one retriever’s cutoff never reaches fusion. If N is very large, downstream work increases and low-quality candidates can make inspection harder. Start with the number of results your application needs, then retrieve a larger candidate pool from each path and measure whether relevant items are being missed before fusion.

Use query slices when reviewing results. Exact-match queries such as error codes, SKUs, and names should expose the value of BM25. Paraphrased questions and conceptual queries should expose the value of dense retrieval. Also inspect disagreements: documents returned by only one path frequently reveal missing metadata, poor chunk boundaries, vocabulary gaps, or embedding-model limitations.

  • Log dense rank, BM25 rank, and fused rank for returned documents.
  • Create test sets for exact terms, paraphrases, mixed queries, and short ambiguous queries.
  • Tune k and candidate depth using judged relevance examples.
  • Apply filters consistently to both retrieval paths before fusion when possible.