Why raw dense and BM25 scores should not be added casually

Dense retrieval ranks documents by semantic proximity in an embedding space. BM25 ranks documents from term occurrence and document statistics. Both outputs are useful, but a score from one system does not automatically have the same scale or meaning as a score from the other.

Adding those values with a fixed weight can make ranking sensitive to score distributions rather than relevance. A query with unusually concentrated dense scores, for example, may behave differently from one where BM25 produces a sharp lexical match. The issue is not that either signal is wrong; it is that the signals require a deliberate common representation before they are combined.

  • Dense search can surface semantically related wording.
  • BM25 can preserve exact terms, identifiers, and rare phrases.
  • Raw score ranges may vary across retrievers and queries.
  • A fusion method should reduce dependence on incompatible scales.

Use reciprocal rank fusion as a simple common language

Reciprocal rank fusion (RRF) combines result lists using document positions. For each document, add a contribution based on its rank in each list: 1 divided by a constant plus that rank. Documents that appear near the top of one or both lists rise in the merged ranking, without requiring dense and BM25 scores to be calibrated against each other.

For an API-first retrieval workflow, request a candidate list from dense search and another from BM25, retain document identifiers and ranks, then merge on the identifier. If a document appears in both lists, it receives two contributions. If it appears in only one, it can still be returned when that rank is strong enough.

  • Choose a candidate depth for each retrieval path before fusion.
  • Assign ranks starting consistently, such as rank 1 for the first result.
  • Compute: fused_score = Σ 1 / (k + rank).
  • Sort descending by fused score and fetch the final document payloads.

Make fusion observable before making it sophisticated

Start by logging which retrieval path contributed each final result. For every query, record the dense rank, BM25 rank, fused rank, and whether the document was present in one or both candidate sets. This makes it possible to distinguish a semantic recall issue from a lexical recall issue instead of treating hybrid retrieval as a black box.

Then build a small evaluation set from real query patterns. Include exact identifiers, short keyword queries, natural-language questions, and queries containing terminology that users may phrase in several ways. Review whether the final list contains the expected documents and inspect the contribution pattern when it does not.

RRF is a baseline, not a claim that all queries need equal dense and sparse influence. Once its behavior is understood, application-specific rules can be evaluated carefully—for example, protecting an exact identifier match or changing candidate depths for a known query class. Keep those rules explicit and testable rather than embedding them in unexplained score arithmetic.

  • Log per-result ranks from dense search and BM25.
  • Evaluate representative query classes, not only natural-language questions.
  • Inspect misses by candidate generation before changing fusion.
  • Treat exceptions for identifiers or filters as explicit retrieval policy.