Why raw dense and BM25 scores should not be added

Dense retrieval ranks records by the relationship between a query embedding and stored embeddings. BM25 ranks records from term-level evidence, including matching query terms and their distribution in a document. Both produce useful rankings, but their numeric scores are not automatically comparable.

Adding those scores directly creates an implicit assumption: that a one-point change in one system means the same thing as a one-point change in the other. That assumption may fail as corpus composition, query length, embedding models, or search settings change.

  • Treat each retriever's output first as an ordered list, not as a shared scoring scale.
  • Use a stable document or chunk identifier so results from both paths can be joined.
  • Keep the original per-retriever rank and score for debugging, even if fusion uses only rank.

Fuse candidate lists with reciprocal rank fusion

RRF assigns each result a contribution based on its position in a ranked list. For a document d, the fused score can be written as: RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of d from retriever i and k is a positive constant.

A record appearing near the top of either the S3 Vectors dense list or the Quickwit BM25 list receives a meaningful contribution. A record supported by both lists receives contributions twice. This makes RRF a useful default when semantic matches and exact-term matches are both important.

  • Retrieve a candidate list from dense search and another from BM25 search.
  • Assign ranks starting at 1 within each list.
  • Sum the RRF contributions by canonical record ID.
  • Sort by fused score and return the top fused candidates.

Make the fusion layer observable and tunable

Implement fusion as a small, deterministic application-layer step after retrieval. Record which source contributed to each returned result, its rank in that source, and its final fused score. This makes it possible to inspect whether a result won through semantic similarity, lexical overlap, or agreement between both.

The candidate depth and the RRF constant are retrieval policy choices, not universal constants. Evaluate them with representative queries and relevance judgments when available. Include queries with identifiers, unusual terminology, paraphrases, and short ambiguous phrases so the evaluation reflects the reasons to run both retrieval paths.

  • Use the same filtering and access-control rules for both retrieval paths before fusion.
  • Deduplicate by the identifier representing the unit users should see, such as a document or chunk.
  • Log source ranks for sampled queries to diagnose unexpected results.
  • Version the retrieval policy so changes to candidate depth or fusion settings can be evaluated clearly.