Why dense and sparse scores should not be added directly

A dense search score expresses similarity in an embedding space. A BM25 score is based on term frequency, document frequency, and field-length normalization. Even when both systems return a numeric value, the numbers do not necessarily have comparable ranges, distributions, or meanings.

Adding raw scores can therefore make one retrieval method dominate for accidental reasons. A query containing a precise product code may need BM25 to lead, while a question phrased in different language from its source material may benefit from dense retrieval. A fusion method should preserve both signals without requiring score values to be directly comparable.

  • Use sparse retrieval for exact wording, identifiers, and rare terms.
  • Use dense retrieval for semantic similarity and paraphrased queries.
  • Avoid assuming a score of 0.8 from one system is equivalent to 0.8 from another.
  • Evaluate fusion using representative queries rather than score intuition alone.

Fuse ranked lists with Reciprocal Rank Fusion

RRF combines result lists by rank rather than raw score. For each document, add a contribution from every list in which it appears: 1 divided by k plus the document's rank. The constant k reduces the impact of very high positions and keeps the method stable when lists have different depths.

For a document d, a common form is RRF(d) = Σ 1 / (k + rank_i(d)). If d is ranked first by dense search and tenth by BM25, it receives credit from both placements. A document that appears near the top of both lists naturally rises, while a strong result from only one method can still remain competitive.

  • Retrieve a candidate list from dense search and another from BM25.
  • Deduplicate candidates using a stable document or chunk identifier.
  • Assign each candidate an RRF score based on its position in each list.
  • Sort by the fused score, then apply a deterministic tie-breaker such as document ID.

Make fusion observable and tune it with real query classes

RRF removes the need for raw-score calibration, but it does not remove the need for evaluation. Keep the dense rank, sparse rank, and final fused rank in retrieval logs. These fields make it possible to see whether a result won because both systems agreed or because one retrieval path contributed it.

Start with a fixed k and equal treatment of both ranked lists. Then test query classes separately: exact lookup queries, natural-language questions, short ambiguous searches, and queries containing both prose and identifiers. If one class consistently needs a different balance, adjust candidate depths or apply a deliberate weight to one list rather than silently relying on incompatible score scales.

  • Log the source ranks and fused rank for every selected result.
  • Create a small judged set that includes exact-match and semantic queries.
  • Inspect failures where relevant documents appear in only one candidate list.
  • Tune candidate depth before introducing more complex weighting rules.