Why raw-score blending is fragile

Dense-search scores and BM25 scores are generated by different models and scoring systems. Their numeric ranges, distributions, and behavior across queries need not match. A weighted sum can therefore become dominated by whichever source happens to emit larger values, even when that source is not producing better results for a particular query.

Score normalization can help in some systems, but it introduces further choices: normalize per query or globally, handle outliers, select weights, and monitor score distributions as indexes and embedding models change. For an initial hybrid retrieval design, rank-based fusion is often easier to reason about.

  • Dense retrieval can surface paraphrases and conceptually related passages.
  • BM25 can reward exact identifiers, uncommon terms, and precise wording.
  • Raw score magnitudes alone do not establish comparable relevance.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) = sum of 1 / (k + rank_i(d)) across the lists where d appears. The rank starts at 1, and k is a positive constant that reduces the difference between nearby ranks.

Run dense retrieval against the S3 Vectors-backed path and sparse retrieval through the Quickwit BM25-backed path. Take a bounded candidate list from each, deduplicate by a stable document or chunk identifier, calculate the fused score, and sort descending. A result that ranks well in both lists rises naturally, while a strong result from only one method can still be retained.

  • Use the same document or chunk identifier in both result lists.
  • Choose a candidate depth before fusion so each retriever can contribute.
  • Keep k explicit in configuration rather than burying it in application code.
  • Preserve source ranks for debugging and evaluation.

Make fusion observable and evaluate it by query type

Store the dense rank, BM25 rank, fused rank, and retrieval source for returned candidates. This makes it possible to inspect why a document appeared: it may have broad support from both methods, or it may have been rescued by one retrieval mode. That distinction is useful when investigating poor answers or changing chunking and metadata strategies.

Evaluate hybrid retrieval with a query set that reflects production language. Include exact-name queries, error messages, short conceptual questions, and longer natural-language requests. Compare dense-only, sparse-only, and fused rankings using relevance judgments or downstream task success. RRF is not a substitute for evaluation; it is a robust baseline that avoids premature score-calibration assumptions.

  • Segment results by query class instead of relying only on an overall average.
  • Inspect queries where dense and sparse results have little overlap.
  • Version the embedding model, chunking policy, and fusion configuration with evaluation results.
  • Re-evaluate after corpus changes that alter terminology or document structure.