Why raw score fusion is fragile
Dense retrieval commonly produces a similarity-oriented score derived from vector comparisons. BM25 produces a lexical relevance score based on term occurrences and document statistics. Even when both systems return a number called a score, the numbers are not automatically on a shared scale.
Adding or averaging raw scores can therefore make one retrieval method dominate for reasons unrelated to relevance. A better initial design is to preserve each retriever’s ordering and combine rankings only after each system has independently selected its top candidates.
- Dense search can surface semantically related wording.
- BM25 can strongly reward exact terms, identifiers, and uncommon phrases.
- A score of 0.8 from one retrieval method does not inherently equal 0.8 from another.
- Rank-based fusion avoids requiring direct score comparability.
Apply Reciprocal Rank Fusion to two result lists
RRF assigns each document a contribution based on its position in a result list. For a document at rank r, the contribution is 1 divided by k plus r. Sum that contribution across the dense and BM25 lists, then sort documents by the resulting total.
The constant k reduces the difference between neighboring ranks near the top of a list. It also keeps a document that appears in both lists competitive, even if it is not ranked first by either one. Choose k deliberately, record it with retrieval configuration, and evaluate changes against representative queries.
- Retrieve a bounded candidate set from S3 Vectors.
- Retrieve a bounded candidate set from Quickwit BM25.
- Deduplicate candidates using a stable document or chunk identifier.
- Compute RRF scores, sort descending, and pass the fused list to the next stage.
Make fusion observable before making it complex
Log the dense rank, sparse rank, and fused rank for every returned item. These fields make it possible to inspect why a result appeared and to identify queries where only one retrieval path contributed useful candidates. They are also more actionable than a single final score.
Start with a small evaluation set containing exact-name queries, paraphrased questions, domain terminology, and ambiguous requests. Review whether the fused results retain BM25’s precision for literal matches while gaining dense retrieval’s coverage for semantic phrasing. Only introduce weighting or query-specific routing after the baseline behavior is understood.
- Track which retrieval paths contributed to each fused result.
- Inspect overlap between dense and sparse candidate lists.
- Evaluate failures by query type, not only aggregate relevance.
- Version retrieval settings alongside indexes and application releases.