Why raw dense and sparse scores should not be added

A dense-search similarity score and a BM25 score come from different ranking systems with different distributions. Their numeric ranges, sensitivity to query length, and response to document length may differ. Adding them directly creates an implicit assumption that a one-point increase in either score means the same thing.

That assumption is fragile. A tuning change in the embedding pipeline, index configuration, or BM25 settings can alter score distributions even when the quality of each individual ranking remains useful. A fixed weighted-sum formula may then change behavior for reasons that are difficult to diagnose.

Ranks are easier to compare than raw scores. RRF uses a document’s position in each ranked list rather than the original retrieval score, making it a useful baseline when dense and sparse scores have not been calibrated together.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful when exact query terms carry important meaning.
  • Raw-score fusion requires calibration that may drift as retrieval systems evolve.
  • Rank-based fusion avoids assuming that score scales are equivalent.

Fuse two candidate lists with reciprocal rank fusion

Run the same user query through the dense retrieval path and the BM25 retrieval path. Keep a bounded candidate list from each path, then merge documents by a stable document identifier. For every occurrence of a document, add an RRF contribution based on its rank in that list.

The common formula is RRF(d) = Σ 1 / (k + rankᵢ(d)), where rank 1 is the top result in a source list, k is a positive constant, and the sum covers the lists in which document d appears. Sort the merged documents by the resulting RRF score.

The constant k softens the difference between nearby ranks. The important operational decision is not the exact constant but consistency: define rank indexing clearly, deduplicate before presenting results, and retain source-rank metadata so fused rankings can be inspected.

  • Retrieve top N candidates from S3 Vectors and top N candidates from Quickwit BM25.
  • Use the same canonical document or chunk ID in both result sets.
  • For each list, assign ranks beginning at 1.
  • Sum 1 / (k + rank) for each document and sort descending.

Make fusion observable before making it more complex

Log the dense rank, sparse rank, and fused rank for the documents shown to users. Those fields answer practical questions during review: did an exact-match document enter through BM25, did a semantically related result enter through dense retrieval, and did duplication across both lists raise a result appropriately?

Evaluate with a query set that reflects real retrieval work. Include queries with identifiers, acronyms, quoted phrases, natural-language questions, short ambiguous requests, and vocabulary that differs from the wording in the target documents. Review the top results rather than relying only on aggregate impressions.

RRF is a baseline, not a claim that every query needs both signals equally. If later evidence supports query-aware routing, source weighting, reranking, or score calibration, introduce one change at a time and compare it against the rank-fusion baseline.

  • Record source membership and source rank for every fused result.
  • Inspect failures by query type, especially exact-term and paraphrase-heavy queries.
  • Watch for duplicate or inconsistent identifiers across indexes.
  • Keep a simple RRF configuration as a regression baseline.