Why raw-score blending is fragile

A tempting hybrid-retrieval design is to add a dense-search score to a BM25 score. This is usually unreliable. A dense similarity value and a BM25 relevance value are produced by different models and scoring systems, so their ranges and distributions need not be comparable.

Even score normalization can be unstable. The score distribution for a short query with a unique product code may look very different from the distribution for a broad natural-language question. A fixed dense-versus-sparse weight can therefore behave differently from one query to the next.

  • Dense retrieval is useful when query and document wording differ but their meaning is related.
  • BM25 is useful for literal matches, rare terms, error codes, names, and identifiers.
  • Score magnitudes are not automatically relevance probabilities.
  • Rank positions are easier to compare across retrieval methods than raw scores.

Apply Reciprocal Rank Fusion to the result lists

RRF assigns each document a contribution based on its position in each ranked list, then sums those contributions. For a document d, a common formula is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of d in list i and k is a positive constant.

The constant k reduces the difference between nearby positions near the top of a list. A document that appears highly in both dense and BM25 results receives a strong combined signal. A document found by only one method can still appear, which preserves useful recall for queries that favor one retrieval mode.

  • Retrieve a bounded candidate list from dense search and another from BM25.
  • Use a stable document identifier to deduplicate candidates across lists.
  • Treat missing documents as contributing zero from that retrieval path.
  • Sort documents by their summed RRF scores and return the top combined set.

Make fusion observable and tune it with judged queries

Start with an evaluation set that reflects real traffic: exact lookup queries, conceptual questions, mixed terminology, and queries with filters. For every query, record the dense rank, BM25 rank, fused rank, and whether the selected result was relevant. This makes it possible to diagnose whether a miss came from candidate generation or from the fusion step.

Tune candidate depth and the RRF constant against judged results rather than intuition. Candidate lists that are too shallow can prevent a relevant document from reaching fusion at all, while excessively deep lists can add noise and operational work. Keep metadata filters and access-control constraints consistent for both retrieval paths before merging results.

  • Log per-query membership and rank from both result lists.
  • Compare dense-only, BM25-only, and fused results on the same judged query set.
  • Use the same filtering rules before fusion to avoid merging ineligible documents.
  • Review zero-result and low-confidence queries separately; fusion cannot recover content that was never indexed.