Why raw score addition is fragile

Dense-search similarity and BM25 scores are produced by different retrieval models. Their numeric ranges, distributions, and sensitivity to query length can differ substantially. A score of 0.8 from one system does not inherently carry the same meaning as a score of 0.8, 8, or 80 from another.

Adding raw scores can therefore make one retrieval path dominate because of scale rather than relevance. This may be especially visible when a query contains a rare identifier, product name, or error code: sparse retrieval may surface exact matches that semantic search ranks lower, while a natural-language question may benefit more from dense retrieval.

  • Do not assume dense and BM25 scores share a common scale.
  • Inspect result overlap as well as individual scores.
  • Keep the dense and sparse candidate lists available for debugging.
  • Evaluate representative query classes, not only an average query.

Use reciprocal rank fusion as a stable baseline

Reciprocal rank fusion (RRF) avoids direct comparison of raw scores. Instead, it awards a document points based on its position in each ranked list. A document appearing near the top of both the S3 Vectors dense results and the Quickwit BM25 results receives a stronger combined signal than a document appearing in only one list.

For each candidate document d, calculate a fused score as the sum of 1 divided by k plus its rank in each list. The constant k reduces the impact of small rank differences at the top of a list. The exact value should be selected through evaluation, but the important property is that fusion operates on rank, not incompatible score magnitudes.

  • Retrieve a bounded candidate set from dense search and BM25.
  • Assign ranks independently within each result list.
  • Sum reciprocal-rank contributions for document IDs found in either list.
  • Sort by fused score, then return the top results to the application.

Make fusion observable and query-aware

A fusion method should be easy to inspect. Log the final rank, dense rank, sparse rank, and fused score for returned documents. These fields make it possible to explain why a result appeared and to identify cases where one retrieval method consistently contributes little value.

After establishing an RRF baseline, segment evaluation by query intent. Exact lookup queries, such as identifiers and quoted names, may need a larger sparse candidate pool. Conceptual questions may need broader dense candidates. This is a retrieval-policy decision at the application layer, not a reason to force two fundamentally different score systems into a single uncalibrated number.

  • Record per-source ranks alongside the merged ranking.
  • Test identifier, keyword-heavy, and natural-language queries separately.
  • Review zero-overlap queries; they often reveal coverage gaps.
  • Change one variable at a time: candidate depth, k, or query routing.