Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are not automatically comparable. Their ranges, distributions, and meanings depend on each retrieval method, its configuration, the query, and the indexed collection. Adding them directly can make one retriever dominate simply because its scores have a larger numerical range.

Normalizing scores can help in controlled settings, but it introduces another system to maintain. A normalization rule that looks sensible for short natural-language questions may behave differently for identifier-heavy queries or documents with uneven chunk lengths. Rank-based fusion avoids assuming that either backend's score is a universal measure of relevance.

  • Keep the dense and sparse retrieval paths independently useful.
  • Use a shared document or chunk identifier across both indexes.
  • Treat each backend's output as an ordered candidate list.

Fuse candidate lists with reciprocal rank fusion

RRF assigns each item a contribution based on its position in each result list. For a document d, the fused score is: RRF(d) = Σ 1 / (k + rankᵢ(d)). Sum across the dense and BM25 lists in which the document appears. The constant k reduces the difference between adjacent top ranks and prevents the first position from overwhelming all other evidence.

For example, if the same chunk ranks third in dense search and seventh in BM25, it receives contributions from both lists. A chunk found only by BM25 can still rank well when it appears near the top of that list. This makes fusion useful when exact lexical evidence and semantic similarity each uncover relevant but partially different candidates.

  • Request a candidate depth from both dense and sparse retrieval before fusion.
  • Use 1-based ranks: the first result has rank 1.
  • Deduplicate results by the shared chunk or document ID before returning them.
  • Apply any final business filters consistently to both candidate lists.

Tune retrieval depth with real query sets

Candidate depth matters because RRF can only promote an item that at least one retriever returned. If either list is too short, useful overlap and complementary results never reach the fusion stage. Start with equal depths for dense and BM25, then inspect whether relevant results are commonly present just beyond the cutoff.

Evaluate with queries that reflect production traffic rather than only polished examples. Include paraphrased questions, product names, acronyms, quoted error text, identifiers, and short ambiguous queries. Review not only whether a relevant result appears, but also whether the top few returned chunks provide enough context for the next stage of the application.

  • Log dense rank, BM25 rank, and fused rank for returned candidates.
  • Compare dense-only, BM25-only, and fused result sets on the same queries.
  • Inspect failures where a relevant chunk is absent from both candidate lists.
  • Version fusion parameters so ranking changes are traceable.