Why raw-score blending is fragile

It is tempting to assign a weight to a dense-search score, assign another weight to a BM25 score, and add them together. That approach assumes the two scores have comparable meanings and ranges. In practice, they often do not: a vector similarity score and a BM25 score are produced by different ranking models and can change distribution as the corpus, query, or configuration changes.

Score normalization can help in some controlled settings, but it introduces another set of choices: which normalization method to use, what population of results defines the distribution, and how to handle short or unusual result lists. These choices can make ranking behavior harder to reason about operationally.

  • Dense search favors semantic proximity between a query and document representation.
  • BM25 rewards term matches using corpus statistics and document-length effects.
  • The same numeric value from two retrieval systems should not automatically be treated as equivalent evidence.

Fuse ranks, not scores, with RRF

Reciprocal rank fusion combines result lists using position rather than each engine's raw score. For every document returned by a retriever, add 1 divided by k plus its rank. The final RRF score is the sum of those contributions across lists. A document that appears near the top of both dense and sparse results receives stronger support than one that appears only once at a low rank.

The constant k reduces the impact of small rank differences at the very top of a list. The important implementation detail is consistency: use one-based ranks, deduplicate by a stable document identifier, and retain retrieval metadata so that the final ranked result can be inspected later.

  • Retrieve a bounded candidate list from S3 Vectors and a bounded candidate list from Quickwit BM25.
  • Represent each result as a stable document ID, rank, source, and optional source-specific score.
  • For each ID, calculate RRF = Σ 1 / (k + rank).
  • Sort descending by RRF score and return the top results needed by the application.

Make hybrid retrieval observable and testable

RRF is simple, but it still has parameters and operational assumptions. Candidate depth determines which documents can participate in fusion, while k controls how sharply rank position affects contribution. Treat both as configuration values and evaluate them against a representative query set rather than assuming a single setting fits every corpus.

Logging the component rankings is as important as logging the final list. When a result appears unexpectedly, engineers should be able to see whether it was introduced by dense retrieval, BM25 retrieval, or agreement between both. This record is also useful when documents are missing, IDs are inconsistent across indexes, or one retrieval path returns fewer candidates than expected.

  • Use the same canonical document ID in dense and sparse indexes.
  • Record candidate counts and ranks from each retrieval path before fusion.
  • Include exact-token queries, semantic paraphrases, mixed queries, and identifier-heavy queries in evaluation.
  • Apply filters consistently before or during retrieval so fused candidates represent the same searchable scope.