Why dense and sparse scores should not be added directly

A dense search score represents similarity in an embedding space. A BM25 score represents lexical evidence based on term frequency, document frequency, and document length. Even when both lists are useful, their numeric values do not share a common interpretation.

Adding raw scores can make one retriever dominate simply because its score range is wider. Normalizing scores can help in controlled experiments, but it introduces choices about distributions, outliers, query classes, and evaluation data. Rank-based fusion avoids treating unlike scores as if they were measurements on the same scale.

  • Dense retrieval can surface paraphrases and conceptually related passages.
  • BM25 can favor exact identifiers, rare terms, filenames, error codes, and quoted language.
  • Raw score magnitudes are retriever-specific and should not be assumed comparable.
  • A fusion method should preserve useful candidates from both result lists.

Fuse candidate lists with reciprocal rank fusion

RRF assigns each document a contribution based on its rank in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across the lists in which d appears. The constant k reduces the difference between adjacent top ranks and prevents the first few positions from overwhelming all other evidence.

For example, retrieve the top N dense results from S3 Vectors and the top N BM25 results from Quickwit. Normalize document IDs, deduplicate the union, calculate each document's RRF score, and sort descending. A document appearing in both lists generally rises, while a strong result unique to either retriever remains eligible.

  • Use one-based ranks: the first result has rank 1.
  • Choose the same candidate depth for both lists initially, then evaluate changes deliberately.
  • Use a stable document or chunk ID as the deduplication key.
  • Keep per-retriever ranks in the response or logs to make fused results explainable.

Make fusion observable before making it complex

Start with a small, representative evaluation set that includes exact-match queries, natural-language questions, acronym-heavy requests, and ambiguous terms. Review whether the relevant chunk appears in either candidate list and whether fusion moves it into the positions your application actually uses.

Operational logging is as important as the formula. Record query metadata, the dense rank, the BM25 rank, the fused rank, and the selected document ID. This makes it possible to distinguish a retrieval miss from a fusion mistake and to identify query patterns where one retrieval mode is consistently more valuable.

  • Inspect candidate overlap: zero overlap is not automatically bad, but it is informative.
  • Measure retrieval quality at the cutoff used by downstream generation or user interfaces.
  • Test changes to k and candidate depth against the same held-out query set.
  • Only introduce query-dependent weights after establishing a clear failure pattern.