Why raw-score blending is a trap

Dense-search similarity scores and BM25 scores are produced by different retrieval methods. Their ranges, distributions, and sensitivity to query length can differ substantially. Adding them directly can make one retriever dominate for reasons that have little to do with result relevance.

This is especially noticeable for mixed workloads. A natural-language question may benefit from dense retrieval, while a query containing a part number, error code, or quoted phrase may depend heavily on lexical matching. A merge strategy should preserve both paths rather than assume their scores are interchangeable.

  • Do not assume a score of 0.8 from one retriever has the same meaning as 0.8 from another.
  • Avoid tuning weights against a single query type or a small hand-picked set of examples.
  • Keep the source retriever and its original rank in every candidate record for debugging.

Start with reciprocal rank fusion

Reciprocal rank fusion (RRF) is a practical baseline because it combines ranks instead of raw scores. For each document returned by a retriever, assign a contribution of 1 divided by k plus its rank. Sum contributions when the same document appears in both result sets, then sort by the total.

The constant k dampens the difference between nearby ranks. A larger value reduces the advantage of being first versus being several positions lower, while still rewarding documents that are consistently near the top. The right value is an application decision, so treat it as configuration and evaluate it with representative queries.

  • Fetch a bounded candidate set from dense search and BM25 before fusion.
  • Use a stable document identifier to deduplicate candidates across result sets.
  • Record dense rank, sparse rank, and fused score alongside the final ordering.
  • Apply filters consistently before fusion whenever the retrieval path supports them.

Make the merge layer observable and easy to revise

Implement fusion as a small deterministic function between retrieval and any later reranking or generation step. Its input can be two ranked lists: one from regional S3 Vectors dense search and one from Quickwit BM25 sparse search. Its output should include the fused ranking and enough metadata to explain why each item was included.

Evaluation does not require a large formal benchmark to be useful. Maintain a growing set of representative queries, including semantic questions, exact-name lookups, acronym-heavy requests, and queries with filters. Review whether useful documents appear in the candidate pool and whether the merge order supports the task users actually need to complete.

  • Log candidate overlap: documents returned by both retrievers are often valuable diagnostic signals.
  • Track empty-result cases separately for dense and sparse paths.
  • Version fusion settings so ranking changes can be tied to a specific configuration.
  • Use qualitative review to identify query classes that need different retrieval or filtering treatment.