Why raw dense and sparse scores should not be added

Dense retrieval ranks documents by vector similarity. Sparse BM25 retrieval ranks them from term-frequency and corpus statistics. Even when both systems return numeric scores, those numbers do not represent a common unit.

Adding a similarity score to a BM25 score can make ranking behavior sensitive to index settings, query wording, embedding changes, or score distributions. A weight that appears reasonable for one query class may behave poorly for another.

The safer starting point is to treat each retrieval method as a ranked signal rather than assuming its scores can be compared directly.

  • Dense search helps with semantic matches and paraphrases.
  • BM25 helps with exact terms, identifiers, and uncommon vocabulary.
  • Raw score ranges can change independently across retrieval systems.

Fuse result lists with reciprocal rank fusion

Reciprocal rank fusion assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus its rank in every list where it appears. The constant k reduces the influence of small rank differences near the top of a list.

For example, retrieve a bounded candidate list from dense search and another from BM25. A document that ranks highly in both lists receives a strong combined score. A document that appears in only one list can still be retained when that retrieval method ranks it well.

A commonly used initial value for k is 60, but it is a tuning parameter rather than a universal rule. Keep the value fixed while evaluating representative queries, then adjust only if the resulting top ranks show a consistent problem.

  • Deduplicate documents by a stable document ID before scoring.
  • Use one-based ranks: first place has rank 1.
  • Sum RRF contributions across dense and sparse lists.
  • Sort by fused score, then use a deterministic tie-breaker such as document ID.

Make the fusion layer observable and testable

Store the component ranks alongside the final result during development: dense rank, BM25 rank, and fused rank. This makes it possible to explain why a document appeared and to distinguish a retrieval issue from a fusion issue.

Build a small evaluation set that includes semantic questions, exact product or error-code lookups, mixed queries, and queries with important filters. Review not only whether a relevant result is present, but whether it is placed early enough for the consuming experience.

RRF is especially useful as a stable baseline because it avoids score calibration. If later requirements demand learned ranking or query-specific weighting, that work can be compared against a transparent rank-fusion baseline.

  • Log candidate list sizes and overlap between dense and BM25 results.
  • Record per-document source ranks for debugging.
  • Evaluate top-k relevance separately for exact-match and semantic query groups.
  • Apply authorization and metadata filters consistently before returning results.