Why raw dense and BM25 scores should not be added

A dense retriever returns a similarity-derived score, while BM25 returns a lexical relevance score based on term frequency, document frequency, and document length. Even when both are useful rankings, their numeric values do not automatically mean the same thing. Adding them directly can let one ranking dominate simply because its score range is larger.

Score normalization can help in controlled settings, but it introduces choices about distributions, query behavior, and calibration data. RRF takes a different route: it uses each document’s position in a result list rather than its raw score. That makes it a useful baseline when combining dense results from regional S3 Vectors with sparse BM25 results from Quickwit.

  • Dense retrieval helps with semantic similarity and paraphrases.
  • BM25 helps preserve exact-token matching for names, codes, and quoted phrases.
  • A rank is easier to compare across retrievers than an uncalibrated relevance score.

Fuse two ranked lists with RRF

For every candidate document returned by either retriever, calculate an RRF score by summing 1 divided by k plus the document’s rank in each list. In notation: RRF(d) = Σ 1 / (k + rank_i(d)). A document appearing near the top of both lists receives more credit than one appearing only once near the bottom.

Choose a shared candidate depth before fusion. For example, retrieve the top N documents from dense search and the top N documents from BM25, deduplicate by a stable document or chunk identifier, compute the fused score, and sort descending. The constant k softens the advantage of the first few ranks; treat it as a tuning parameter rather than a universal answer.

  • Use one stable identifier for deduplication across dense and sparse indexes.
  • Keep the original dense rank and BM25 rank for debugging.
  • Retrieve enough candidates to allow overlap, but avoid expanding the candidate set without a reason.
  • Apply metadata filters consistently to both retrieval paths when the query requires them.

Evaluate the fused ranking on real query classes

RRF is not a replacement for evaluation. Build a small query set that reflects the requests your application receives: natural-language questions, product names, short keyword searches, identifiers, and misspellings where relevant. Record whether the desired document or chunk appears in the final shortlist, not only whether either individual retriever found it.

Inspect disagreements as carefully as successes. If BM25 consistently wins for identifier-heavy queries, that is useful evidence, not a failure of dense search. If dense retrieval finds relevant wording that lexical search misses, preserve that benefit. The goal is a merged ranking that handles varied query intent predictably, with a design that remains understandable when results need investigation.

  • Measure recall at the candidate cutoff before judging final-answer quality.
  • Compare dense-only, BM25-only, and RRF results for the same labeled queries.
  • Log rank positions and source membership for each fused candidate.
  • Revisit candidate depth and k after collecting representative relevance judgments.