Why raw-score blending is fragile

A dense-search score and a BM25 score are not inherently comparable. Their ranges, distributions, and meanings depend on the underlying retrieval method, the index, the query, and configuration choices. Adding them directly can make one retriever dominate simply because its scores occupy a larger numerical range.

Score normalization can help, but it introduces its own operating assumptions. A min-max transform depends on the candidate set returned for each query, while a global normalization strategy needs representative score distributions and can drift as the corpus changes. For an initial hybrid retrieval path, rank-based fusion is often easier to reason about.

  • Dense search emphasizes semantic proximity.
  • BM25 emphasizes query-term evidence and term rarity.
  • A score of 0.8 from one retriever does not automatically mean more than a score of 12 from another.
  • Rank order is usually a safer shared signal than raw score magnitude.

Fuse result lists with reciprocal rank fusion

Reciprocal rank fusion (RRF) assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across the retrieval lists where it appears. The constant k reduces the difference between adjacent top ranks and prevents a single first-place result from overwhelming all other evidence.

In practice, retrieve a bounded candidate list from S3 Vectors and another from Quickwit BM25, map both lists to stable document identifiers, compute RRF per identifier, then sort by the fused value. A document that appears near the top of both lists rises naturally, while a document found by only one retrieval method can still remain eligible.

  • Use one-based ranks: the first result has rank 1.
  • Deduplicate by a stable document or chunk identifier before returning results.
  • Keep the original dense and BM25 ranks as diagnostic metadata.
  • Choose candidate depths large enough to create useful overlap, then validate with representative queries.

Make fusion observable before tuning it

RRF avoids direct score calibration, but it is still a retrieval policy that should be inspected. Log which source contributed each returned result, its rank in each source, its fused score, and the query class when available. These fields make it possible to see whether exact-match queries are being supported by BM25 and whether paraphrased queries are receiving meaningful dense candidates.

Start with a fixed k and equal contribution from both lists, then evaluate against a small judged query set drawn from real usage patterns. If one source is consistently more reliable for a known query class, apply source weights deliberately rather than silently relying on score scale. Revisit the judgment set when chunking, embeddings, analyzers, or corpus composition changes.

  • Include identifier-heavy, natural-language, and mixed queries in evaluation.
  • Inspect failures where relevant documents appear in only one candidate list.
  • Track source overlap; zero overlap is a useful signal, not automatically an error.
  • Treat fusion configuration as versioned retrieval logic.