Why raw dense and BM25 scores should not be added

A dense retrieval score and a BM25 score are produced by different ranking systems. Their numeric ranges, distributions, and meanings can vary by query, corpus, embedding model, and BM25 configuration. A score of 0.8 from one retriever is not inherently equivalent to a score of 8 from another.

Directly adding or weighting raw scores can therefore create brittle behavior. A small configuration change may cause one retriever’s numeric scale to dominate the other, even when both are finding useful documents. Rank-based fusion avoids requiring those scores to be calibrated against each other.

  • Dense search retrieves by vector similarity from regional S3 Vectors.
  • Sparse search retrieves lexical matches using Quickwit BM25.
  • Each path can return its own top-k ranked document identifiers.
  • Fusion should operate on a stable identifier shared by both result lists.

Fuse ranked lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) for every list where d appears. The constant k reduces the impact of a single top position and makes the method less sensitive to minor rank changes.

For example, if a document ranks second in dense retrieval and fifth in BM25, it receives contributions from both placements. A document found by only one retriever can still rank well, but documents supported by both signals are naturally promoted.

  • Retrieve a bounded candidate list from dense search and another from BM25.
  • Use one-based ranks: first result has rank 1.
  • Choose and document a fixed RRF constant k for an experiment or release.
  • Sort documents by descending fused score, then apply a deterministic tie-breaker.

Make fusion observable and test it with query slices

Store enough retrieval metadata to explain why a document was returned: whether it came from dense retrieval, BM25, or both; its rank in each list; and its final fused rank. This record is more useful for debugging than a fused score alone because it exposes the source of a ranking change.

Evaluate RRF on slices of real queries rather than relying on a single aggregate impression. Exact product codes, quoted phrases, acronyms, and rare identifiers are useful sparse-heavy slices. Natural-language questions, paraphrases, and terminology variation are useful dense-heavy slices. The goal is not to force identical behavior across slices, but to verify that fusion preserves useful evidence from both paths.

  • Log candidate counts and rank positions for both retrievers.
  • Check whether relevant documents appear in one list, both lists, or neither.
  • Inspect queries where fused top results differ materially from each individual retriever.
  • Version the fusion rule alongside retrieval and indexing configuration.