Why raw score addition is fragile

A dense-search score and a BM25 score are produced by different retrieval models. Their ranges, distributions, and sensitivity to query length are not inherently aligned. Adding them directly can cause one retriever to dominate simply because its numerical scale is larger.

Score normalization can help, but it introduces operational choices: which statistics to collect, how often to refresh them, and how to handle different document collections or changing query traffic. A rank-based method avoids making raw scores comparable in the first place.

  • Dense retrieval ranks semantic proximity in embedding space.
  • BM25 ranks term-based relevance using corpus statistics.
  • A score of 1.2 from one path has no universal relationship to 1.2 from the other.
  • Ranking positions are easier to combine than unrelated score scales.

Use reciprocal rank fusion for a stable baseline

Reciprocal rank fusion, commonly abbreviated RRF, assigns a contribution to each document based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank across the lists in which it appears: RRF(d) = Σ 1 / (k + rank_i(d)).

The constant k reduces the impact of small differences near the top of a list. Rather than rewarding a document because one system emitted a large raw score, RRF rewards documents that rank well in dense results, sparse results, or both. It is therefore a useful baseline for combining results returned by S3 Vectors and Quickwit BM25.

  • Fetch the top N candidates from the dense path and the sparse path.
  • Use a stable document identifier to deduplicate candidates across lists.
  • Assign ranks starting at 1 for each result list.
  • Compute an RRF score per unique document, then sort descending.

Choose candidate depth and inspect disagreement

Candidate depth matters because fusion can only promote documents that appear in at least one input list. Start with a bounded top-N from each retriever, then evaluate whether relevant documents are being excluded before fusion. The appropriate depth depends on corpus size, query patterns, and the latency budget of the application.

Review queries where the two retrievers disagree. Exact error messages and codes may appear high in BM25 but low in dense retrieval; paraphrased questions may show the opposite pattern. Those examples are more actionable than a single aggregate metric because they reveal whether the issue is tokenization, document chunking, embeddings, metadata filters, or the query itself.

  • Log each candidate’s dense rank, BM25 rank, and fused rank.
  • Keep query-time filters consistent across both retrieval paths.
  • Test identifier-heavy, natural-language, and mixed queries separately.
  • Treat RRF parameters and candidate depth as application-level configuration to validate with relevance judgments.