Why raw dense and sparse scores should not be added

Dense retrieval ranks documents by semantic proximity between vectors. BM25 ranks documents from term-frequency and corpus-statistics signals. Even when both systems return numeric scores, those numbers have different meanings, ranges, and distribution shapes.

Adding the scores with a fixed weight can make ranking sensitive to query type and implementation details. A score scale can also shift after an index refresh, a change in embedding model, or a corpus change. Rank-based fusion instead uses the ordering each retriever produces.

  • Dense search is useful when relevant wording differs from the query.
  • BM25 is useful when exact terms, identifiers, names, or error codes matter.
  • Ranks are ordinal signals, so they do not require score normalization across retrieval methods.

Apply reciprocal rank fusion to two candidate lists

For a query, request a top-K ranked list from the dense path and a top-K ranked list from the BM25 path. Give every result a stable document identifier shared by both indexes. Then assign each document a fused score based on where it appears in either list.

The standard RRF formula is score(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in retrieval list i. The constant k reduces the influence of tiny rank differences near the top of a list. Documents returned by both retrievers receive contributions from both ranks.

  • Use the same canonical document ID in dense and sparse records.
  • Choose a candidate depth K that leaves enough documents for downstream filtering and reranking.
  • Treat a missing document in one list as contributing zero from that list.
  • Keep the retrieval lists and fusion inputs available in logs for relevance debugging.

Make fusion reliable with consistent retrieval boundaries

Fusion is most useful when the two retrieval paths operate over the same logical document population. Apply equivalent tenant, access-control, language, content-status, and time-window constraints before merging results. Otherwise, a document can receive a rank advantage simply because one path searched a broader corpus.

Start with RRF as a transparent baseline, then evaluate it using representative queries. Inspect whether dense retrieval recovers paraphrases and whether BM25 preserves exact-match intent. If a later reranker is added, use fused results as its candidate set rather than assuming either retriever alone supplies every relevant document.

  • Define filters once and pass equivalent constraints to both retrieval paths.
  • Deduplicate by canonical document ID before returning results.
  • Record dense rank, BM25 rank, and fused rank for each selected result.
  • Review failures by query class, especially product names, acronyms, and natural-language questions.