Why raw dense and BM25 scores should not be added

A tempting hybrid-ranking design is to query both systems and add their scores. That approach is fragile because a dense similarity score and a BM25 score do not necessarily share a scale, distribution, or meaning. A score that is large in one retrieval system is not automatically stronger evidence than a smaller score from the other.

Score ranges can also vary across queries. An exact product code may produce a sharply concentrated BM25 ranking, while a broad conceptual question may produce a more gradual dense ranking. Treating these values as directly comparable can cause one retrieval path to dominate for reasons unrelated to relevance.

  • Dense retrieval prioritizes semantic proximity in embedding space.
  • BM25 prioritizes lexical evidence such as matching terms and term rarity.
  • Score calibration requires validation data and ongoing monitoring.
  • Rank positions are often a safer common input than raw scores.

Fuse two candidate lists with reciprocal rank fusion

Reciprocal rank fusion, commonly abbreviated RRF, combines ranked lists rather than their underlying scores. Query regional S3 Vectors for a dense candidate list and Quickwit BM25 for a sparse candidate list. For every document appearing in either list, add a contribution based on its position in each list.

A common formulation is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in result list i and k is a positive constant. A document that ranks well in both lists gains evidence from both paths, while a document found by only one path can still remain competitive.

  • Choose a candidate depth for each retriever, such as the top N results.
  • Deduplicate results using a stable document or chunk identifier.
  • Use one-based ranks when calculating the fusion contribution.
  • Sort documents by fused score, then apply a deterministic tie-breaker.

Make fusion observable before tuning it

Start with the same candidate depth for dense and sparse retrieval, then inspect fused results for representative queries. Include exact-name queries, error messages, abbreviations, natural-language questions, and queries with rare identifiers. The goal is not to prove that either retriever always wins; it is to identify when each contributes useful candidates.

Log the rank and source contribution for each returned document. This makes it possible to distinguish a result supported by both retrieval paths from one introduced only by dense or sparse search. If results need later reranking, retain these signals as metadata rather than discarding them after fusion.

  • Maintain a small, reviewed query set that reflects real traffic patterns.
  • Record dense rank, BM25 rank, fused rank, and document identifier.
  • Investigate queries where relevant documents appear in only one candidate list.
  • Tune candidate depth and the RRF constant against judged relevance, not intuition alone.