Why raw-score blending is fragile

A dense search result is commonly ordered by vector similarity or distance. A sparse result is ordered by a lexical ranking function such as BM25. Even when both systems return a number called a score, that number is produced by different calculations and should not be assumed to share a common scale.

A weighted formula such as `0.5 × dense_score + 0.5 × bm25_score` can therefore behave unpredictably. A small change in embedding model, query length, document field configuration, or sparse index settings can alter one score distribution without changing the other. The blend may then favor a retrieval method for numerical reasons rather than relevance reasons.

  • Dense retrieval helps with semantic matches and paraphrases.
  • BM25 helps with literal terms, acronyms, codes, and rare names.
  • Raw score ranges can vary across queries and retrieval systems.
  • Rank-based fusion reduces dependence on score-scale assumptions.

Apply Reciprocal Rank Fusion to the two result lists

Run dense search against S3 Vectors and sparse search against Quickwit BM25 using the same query context. Request a candidate list from each system, then assign every returned document an RRF contribution based on its position in each list.

For a document `d`, calculate `RRF(d) = Σ 1 / (k + rank_i(d))`, where `rank_i(d)` is the one-based rank in result list `i`. The constant `k` softens the advantage of the very first positions. A document found by both methods receives two contributions; a document found by only one method can still rank well if it appears near the top.

  • Deduplicate candidates by a stable document or chunk identifier.
  • Use one-based ranks: the first result has rank 1.
  • Start with a shared `k` value and validate it with representative queries.
  • Fuse enough candidates to preserve useful overlap, not only the top few results.

Make fusion observable and easy to tune

Store the retrieval provenance alongside each fused result: dense rank, BM25 rank, fused score, and whether the item appeared in one or both lists. This makes relevance investigations concrete. When a result looks surprising, the team can see whether it was driven by semantic similarity, lexical matching, or agreement between both methods.

Evaluate fusion on a small, maintained query set that reflects production traffic. Include exact lookup queries, natural-language questions, ambiguous phrases, terminology-heavy requests, and queries that should return no result. The goal is not to prove that one retriever always wins; it is to identify where combining signals improves the result set.

  • Log per-retriever ranks before applying fusion.
  • Inspect overlap between dense and sparse candidate lists.
  • Keep query text, filters, and chunking choices consistent during comparisons.
  • Use relevance judgments or human review before changing fusion parameters.