Why raw-score blending is fragile

A dense-search score and a BM25 score are produced by different retrieval models and should not automatically be compared directly. Their ranges, distributions, and sensitivity to query length can differ. Even within one retrieval method, score behavior can change as content, analyzers, embeddings, or index configuration evolve.

A formula such as `0.5 × dense_score + 0.5 × bm25_score` therefore hides a major assumption: that both inputs are calibrated and comparable. If that assumption is wrong, one retrieval path can dominate results for reasons unrelated to relevance.

  • BM25 rewards term occurrences using corpus-dependent statistics.
  • Dense search ranks items according to vector similarity.
  • A score value is not inherently a universal relevance probability.
  • Score ranges may shift as indexed content changes.

Fuse ranks with Reciprocal Rank Fusion

Reciprocal Rank Fusion, usually shortened to RRF, combines ranked lists rather than their raw scores. Retrieve a candidate list from regional S3 Vectors and another from Quickwit BM25, then add a small contribution for each document based on its position in each list.

For a document d, the common formula is `RRF(d) = Σ 1 / (k + rank_i(d))`. The sum runs over the retrieval lists in which d appears. The constant k reduces the influence of the very top positions; 60 is a commonly used starting point, not a universal requirement. Documents returned by both systems receive contributions from both lists, while a strong result from only one system can still be retained.

  • Fetch a bounded top-N list from dense retrieval.
  • Fetch a bounded top-N list from BM25 retrieval.
  • Deduplicate candidates using a stable document or chunk ID.
  • Sum RRF contributions and sort descending before returning the final top results.

Make fusion observable and tune it deliberately

Start with the same candidate depth for both retrieval paths, then inspect representative queries before changing weights or cutoffs. Include queries with natural-language intent, exact titles, codes, acronyms, misspellings, and terms that are new or rare in the corpus. These cases reveal where dense and sparse retrieval complement each other.

Log the source ranks for each fused result, not only its final position. That record makes debugging concrete: a result may have won because it ranked highly in both lists, because BM25 found an exact phrase, or because dense retrieval connected paraphrased language. If the product requires a different balance, use weighted RRF by multiplying each list’s contribution, and validate the change against judged queries.

  • Keep the same canonical IDs across dense and sparse indexes.
  • Record dense rank, BM25 rank, and fused rank for inspected queries.
  • Evaluate changes using a fixed set of relevance judgments where possible.
  • Adjust candidate depth, k, or list weights one variable at a time.