Why raw dense and BM25 scores should not be added

A dense-search score expresses similarity in an embedding space. A BM25 score is driven by term occurrences, document frequency, field length, and query terms. Even when both rankings are relevant, a score of 0.8 from one retrieval system does not have a generally transferable meaning relative to a score of 0.8, 8, or 80 from another.

Adding or averaging raw scores therefore creates an accidental weighting scheme. Small changes to embedding models, BM25 configuration, document chunking, or score distributions can change which retrieval method dominates. This can make relevance behavior difficult to reason about and harder to debug.

  • Use dense retrieval for semantic similarity, paraphrases, and concept-level matches.
  • Use BM25 for exact terms, identifiers, error codes, names, and rare vocabulary.
  • Treat each system’s ordered results as the dependable common signal.
  • Avoid score arithmetic unless the scores have been deliberately calibrated and monitored.

Fuse result lists with Reciprocal Rank Fusion

RRF combines lists by awarding a document credit based on its position rather than its raw score. For each result list, a document at rank r receives 1 divided by k plus r. The final RRF score is the sum of that value across the lists in which the document appears.

The constant k reduces the influence of tiny rank differences near the top of a list. A document appearing strongly in both dense and sparse results tends to rise, while a document that ranks highly in only one method can still remain discoverable. The method is especially useful when separate retrieval engines produce rankings with different score semantics.

  • Request a bounded candidate list from dense search and another from BM25 search.
  • Assign ranks starting at 1 within each list.
  • Compute: RRF(document) = Σ 1 / (k + rank).
  • Sort the union of candidate documents by the summed RRF score, then apply any downstream filtering or reranking.

Make fusion observable and test it against real queries

Keep the retrieval paths visible in application logs or evaluation output. For every fused result, record its dense rank, BM25 rank, fused score, and whether it appeared in one or both lists. This makes it possible to explain why an item was retrieved instead of viewing hybrid retrieval as a black box.

Evaluate with query sets that reflect actual failure modes. Include exact identifiers, short ambiguous queries, long natural-language questions, spelling variants, and vocabulary that may be absent from the embedding model’s training distribution. Compare dense-only, BM25-only, and fused rankings using relevance judgments appropriate to the application.

  • Deduplicate candidates using a stable document or chunk identifier before presenting results.
  • Choose candidate-list depths large enough to create useful overlap, then validate the choice with evaluation data.
  • Keep metadata filters consistent across dense and sparse retrieval paths.
  • Use a reranker only after fusion when additional precision is needed and its behavior can be evaluated.