Why raw dense and BM25 scores should not be added directly

Dense retrieval and BM25 produce scores with different meanings. A dense score may reflect a vector similarity measure, while BM25 scores are driven by term frequency, document frequency, and document-length normalization. Even when both are useful for ranking, equal-looking numeric values do not imply equal relevance.

Adding or averaging uncalibrated scores can make one retriever dominate for accidental reasons such as score range, query length, or corpus composition. It also creates a tuning burden: a weight that works for broad natural-language questions may behave poorly for identifier-heavy or quote-like queries.

  • Dense search is often helpful when the wording of a query differs from the wording of relevant content.
  • BM25 is often helpful when exact tokens matter, including product names, error codes, acronyms, and IDs.
  • Score scales can vary by retrieval method and should not be assumed to be interchangeable.

Apply reciprocal rank fusion to two candidate lists

Run dense retrieval against the vector index and BM25 retrieval against the sparse index. Keep a candidate list from each path, then assign each document a fused score based on its rank in each list. With RRF, a document receives 1 divided by k plus its rank for every list in which it appears.

The fused score is calculated as: RRF(d) = sum of 1 / (k + rank_i(d)). The constant k reduces the impact of small rank differences near the top of a list. A document that appears in both result sets gains evidence from both retrieval methods, while a strong one-list result can still remain competitive.

  • Retrieve a bounded candidate set from dense search and BM25, such as the top N results from each.
  • Deduplicate candidates using a stable document or chunk identifier.
  • Use one-based ranks when computing the fusion score.
  • Sort documents by fused score, then apply a deterministic tie-breaker such as document ID.

Make fusion observable before making it more complex

Log the source ranks that contributed to every fused result. For each returned document, record its dense rank, BM25 rank, fused score, and whether it appeared in one or both lists. This makes it possible to inspect whether hybrid retrieval is recovering complementary material or merely repeating the same candidates.

Start with a fixed k and a consistent retrieval depth, then evaluate representative queries from the application. Pay special attention to queries containing rare terms, structured identifiers, short ambiguous phrases, and conceptual questions. If relevance needs additional improvement, use the fused set as a candidate pool for a later reranking stage rather than overfitting score weights prematurely.

  • Track overlap between dense and BM25 candidate lists.
  • Review queries where one retriever contributes most of the final top results.
  • Keep metadata filters consistent across both retrieval paths.
  • Version retrieval settings so ranking changes can be reproduced.