Why raw score merging is usually brittle

A dense-search score and a BM25 score are produced by different models and calculations. Their numeric ranges, distributions, and sensitivity to query length can differ substantially. Adding the scores directly can make one retriever dominate simply because its scoring scale is wider, not because its results are more useful.

Score normalization can help, but it introduces choices about calibration data, outliers, and how frequently to refresh the normalization parameters. Those choices become more difficult when the corpus changes or when query traffic covers several domains.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for lexical matches, rare terms, and identifiers.
  • A score of 0.7 from one retriever is not inherently comparable to 0.7 from another.
  • Rank-based merging avoids treating unlike score scales as equivalent.

Use reciprocal rank fusion as the first merge policy

RRF assigns each document a contribution based on its position in each result list, then sums the contributions. A common form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in retriever i and k is a positive constant that reduces the influence of very high ranks.

In practice, issue a dense query to S3 Vectors and a sparse query to Quickwit, request a bounded candidate list from each, deduplicate by a stable document or chunk identifier, and calculate the fused ranking in the application layer. Documents returned by both paths gain two contributions; documents returned by only one path can still appear when their rank is strong.

  • Use the same logical chunk IDs across dense and sparse indexes.
  • Keep rank positions one-based and define a deterministic tie-breaker.
  • Choose a candidate depth large enough for overlap and useful long-tail results.
  • Start with equal weights; introduce per-retriever weights only after evaluating representative queries.

Make fusion observable before making it clever

Log enough information to explain each final result: whether it came from dense search, BM25, or both; its rank in each list; and its fused score. This makes it possible to spot patterns such as exact-code queries relying almost entirely on BM25 or broad conceptual queries relying mainly on dense retrieval.

Evaluate with a small, maintained set of real query intents rather than only generic relevance examples. Include acronym-heavy queries, quoted phrases, misspellings, natural-language questions, and queries whose answer depends on a specific document version. Review both top-result quality and whether the candidate lists contain the needed result before fusion.

RRF is not a replacement for good indexing. Chunk boundaries, metadata filters, text extraction, embedding choice, and corpus freshness still determine what either retriever can surface. Fusion is most valuable when it combines two healthy retrieval paths with complementary evidence.

  • Record source membership and pre-fusion ranks for every returned result.
  • Measure candidate recall separately from final ranking quality.
  • Test filters consistently across both retrieval paths when the query requires them.
  • Treat changes to chunking or indexing as retrieval changes that need re-evaluation.