Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different retrieval models and have different distributions. A BM25 score depends on term frequency, document frequency, field configuration, and document length. A dense score depends on the embedding model and its similarity function. Even within one system, score ranges can shift as the corpus or indexing configuration changes.

Adding the two values directly creates an accidental weighting scheme. A large numeric range from one retriever can dominate the other even when both result lists are useful. Normalizing scores can help in carefully evaluated systems, but it requires monitoring and assumptions about score distributions. Rank-based fusion is a simpler starting point because it uses the ordering each retriever already produced.

  • Use dense retrieval for semantic similarity and paraphrased intent.
  • Use BM25 for exact phrases, product names, codes, and uncommon terms.
  • Keep dense and sparse scores separate unless they have been calibrated and evaluated together.

Fuse candidate lists with reciprocal rank fusion

Reciprocal rank fusion, or RRF, assigns each document a contribution based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank for every list where the document appears: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the difference between very high ranks and lower ranks.

Run a dense query against the vector index and a sparse query against the BM25 index, then fuse the returned document IDs in the application layer. A document that ranks well in both lists rises naturally. A document that appears in only one list can still be retained, which matters when a query is strongly lexical or strongly semantic.

Choose one stable identifier for every retrievable chunk before indexing. The dense and sparse result sets must refer to the same chunk IDs, or a fusion layer cannot reliably deduplicate and combine candidates.

  • Retrieve a candidate window from both dense and sparse search before fusing.
  • Deduplicate by chunk ID, not by displayed text.
  • Use 1-based ranks when implementing the RRF formula.
  • Start with equal contribution from both lists; add weights only after evaluation shows a need.

Evaluate fusion on the queries your users actually ask

Hybrid retrieval is not automatically better for every query. Build a small evaluation set that includes exact-name lookups, troubleshooting questions, paraphrases, ambiguous questions, and queries containing identifiers. For each query, record which chunk or document should be retrieved and inspect whether dense-only, sparse-only, and fused retrieval place it in the candidate set.

Look beyond a single aggregate metric. If BM25 consistently rescues identifier-heavy queries, that is evidence to preserve sparse candidates. If dense retrieval finds useful terminology variants, preserve its contribution as well. Review failures by query type: a chunking problem, an indexing-field problem, and a fusion problem can look similar in a final answer.

RRF is valuable because it gives teams a transparent baseline. Once that baseline is stable, changes such as query rewriting, metadata filters, retriever weights, or reranking can be tested against it rather than introduced as unmeasured complexity.

  • Track whether a relevant result appears in the top candidate window before generation.
  • Include both semantic and exact-match queries in relevance reviews.
  • Test changes against a fixed query set and inspect regressions.
  • Treat chunk boundaries and metadata quality as part of retrieval quality.