Why raw dense and BM25 scores should not be added

A dense retriever ranks documents by proximity between embeddings. A BM25 retriever ranks them from term frequency, inverse document frequency, and document-length effects. Even when both systems return a numeric score, those values do not represent the same quantity.

Adding the scores directly creates an accidental weighting scheme. A small change in embedding model, index settings, corpus composition, or BM25 configuration can change one score distribution while leaving the other unchanged. The resulting rank order may shift for reasons unrelated to relevance.

Score normalization can help in controlled environments, but it requires ongoing validation. Percentile, min-max, and z-score approaches all depend on assumptions about score distributions. RRF avoids that dependency by retaining only ordinal information: which documents each retriever considered most relevant.

  • Dense retrieval can recover semantic paraphrases and conceptually related content.
  • BM25 is strong when exact terms, identifiers, error codes, and uncommon tokens matter.
  • Raw score magnitudes from the two retrievers should be treated as incomparable by default.

Fuse two candidate lists with RRF

Run the same user query through the dense and sparse paths, then request a candidate list from each. For every document that appears in either list, assign an RRF contribution from each list in which it appears. The document’s final score is the sum of those contributions.

The standard formula is RRF(d) = Σ 1 / (k + rank_i(d)). Here, rank_i(d) is the one-based position of document d in result list i, and k is a positive constant that reduces the impact of very high ranks. Documents returned near the top by both retrievers rise naturally, while a strong result from only one retriever can still remain competitive.

For example, a document ranked second by dense search and eighth by BM25 receives 1/(k+2) + 1/(k+8). A document that appears only at rank one in BM25 receives 1/(k+1). This rewards cross-retriever agreement without requiring either system’s scores to be calibrated.

  • Use one-based ranks consistently: the first result has rank 1.
  • Deduplicate by a stable document or chunk identifier before final sorting.
  • Choose the same candidate depth for both paths initially to make debugging easier.
  • Apply a deterministic tie-breaker, such as document ID, for reproducible responses.

Make candidate depth and evaluation part of the design

RRF can only promote documents that at least one retriever returns. Candidate depth is therefore a recall decision. If dense search returns too few candidates, it may omit a semantically useful document before fusion begins; the same is true for sparse search and exact-match material.

Start with a fixed top-N from regional S3 Vectors and a fixed top-N from Quickwit BM25, fuse the union, then return the application’s requested top-K. Log the source ranks for each returned document. Those rank traces make it possible to see whether a result won because both systems agreed, because dense retrieval surfaced it, or because BM25 did.

Evaluate with representative queries rather than relying on a single query type. Include identifier-heavy searches, natural-language questions, terminology variations, and queries with ambiguous intent. Compare dense-only, BM25-only, and RRF results using judged relevance or downstream task success. RRF is a strong transparent baseline, not a substitute for measurement.

  • Keep source rank and retrieval path metadata with fused results for observability.
  • Test queries containing product names, IDs, acronyms, and natural-language paraphrases.
  • Increase candidate depth when relevant documents are absent from both fused inputs.
  • Use offline relevance judgments before changing fusion constants or adding learned reranking.