Why raw dense and sparse scores should not be added

A dense-search score and a BM25 score are produced by different ranking functions. Their numeric ranges, distributions, and sensitivity to query length can differ. Adding them directly assumes that a one-point change means the same thing in both systems, which is usually an assumption rather than a fact.

This becomes especially fragile when the corpus changes. Adding documents, changing analyzers, selecting a different embedding model, or adjusting retrieval parameters can shift score distributions. A fixed weighted-score formula may then need retuning even when the underlying relevance goal has not changed.

  • Dense retrieval is useful when query and document wording differ but meaning is related.
  • BM25 is useful when exact tokens, identifiers, names, or rare terminology matter.
  • Raw score values are ranking-system-specific signals, not automatically comparable units.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its rank in each result list: 1 divided by k plus the rank. The document’s final score is the sum of its contributions across lists. Here, k is a positive constant that reduces the advantage of appearing at the very first position.

For a query, retrieve a candidate list from regional S3 Vectors and another from Quickwit BM25. Normalize document identifiers before merging, then calculate the RRF score for every document that appears in either list. Sort by the fused score and return the top results.

  • Use one-based ranks: rank 1 is the first result in a list.
  • A document appearing in both lists receives two contributions.
  • A document appearing in only one list can still rank well if it is near the top.
  • Keep the dense and sparse candidate depths explicit so they can be reviewed and adjusted.

Make fusion observable before making it complex

Store retrieval diagnostics alongside the fused result during testing: dense rank, BM25 rank, fused score, and the source lists in which the document appeared. These fields make it possible to distinguish a result supported by both retrievers from one promoted by a single strong signal.

Build a small evaluation set from representative queries, including exact identifiers, short natural-language questions, ambiguous terms, and domain-specific phrases. Review whether RRF preserves exact-match results when needed while also recovering useful semantic matches. Only introduce weighting or query-dependent logic after the baseline behavior is understood.

  • Log query text or a privacy-appropriate query reference, plus per-retriever ranks.
  • Review duplicate and stale document handling before fusion, using a stable canonical document ID.
  • Test queries with identifiers and rare terms separately from conceptual questions.
  • Treat RRF as a clear baseline that can expose where either retrieval path needs improvement.