Why raw-score addition is fragile

A dense retrieval score and a BM25 score are produced by different ranking functions. Their numerical ranges, distributions, and sensitivity to query length can differ substantially. A score of 0.7 from one retrieval path has no universal meaning relative to a score of 12 from another.

Normalizing scores per query can help in controlled settings, but it introduces its own decisions: which normalization method to use, how to handle outliers, and what to do when one result set has nearly identical scores. Those choices can make relevance behavior difficult to reason about and regressions difficult to diagnose.

  • Do not assume score magnitudes represent the same unit of relevance.
  • Avoid choosing fixed dense-versus-BM25 weights solely from a small set of example queries.
  • Keep each retrieval path independently observable before introducing fusion.

Fuse ranked lists with RRF

Reciprocal Rank Fusion uses a document's position in each ranked list, not its raw score. For every document returned by either search path, add 1 divided by k plus its one-based rank. Documents appearing near the top of both lists receive a higher combined score; a strong result from only one list can still remain competitive.

The formula is RRF(d) = Σ 1 / (k + rank_i(d)), where the sum covers the ranked lists in which document d appears. The constant k dampens the difference between nearby positions. A commonly used starting value is 60, but it is a tuning parameter rather than a universal truth.

  • Fetch a bounded candidate list from S3 Vectors and a bounded candidate list from Quickwit BM25.
  • Deduplicate candidates using a stable document or chunk identifier.
  • Assign ranks starting at 1, compute the RRF sum, then sort descending.
  • Return enough candidates from each path that potentially useful overlap is not discarded before fusion.

Make fusion debuggable in production

Store more than the final ranking during evaluation. For each result, retain its dense rank, BM25 rank, fused score, and the retrieval paths that returned it. This makes it possible to distinguish a poor final result caused by candidate generation from one caused by the fusion rule.

Evaluate with query sets that reflect actual search behavior. Include exact identifiers, abbreviated technical questions, natural-language questions, and queries with vocabulary that differs from the source text. RRF is most useful when both retrieval paths contribute distinct relevant candidates, not merely when they produce two versions of the same list.

  • Log candidate counts and overlap between dense and sparse result sets.
  • Inspect zero-overlap queries; they often reveal tokenization, chunking, or embedding issues.
  • Compare a dense-only, BM25-only, and fused baseline on the same judged queries.
  • Tune k and candidate depth separately, changing one variable at a time.