Why raw-score merging is fragile

Dense retrieval ranks documents by the relationship between an embedded query and embedded content. Sparse BM25 retrieval ranks documents according to term matches and corpus statistics. Both are useful signals, but their numeric outputs reflect different scoring models.

Adding the scores directly can make one retrieval path dominate for reasons unrelated to relevance. A score that appears larger is not necessarily stronger; it may simply belong to a scale with a different range, distribution, or response to query length.

  • Dense search can recover semantically related language that does not share exact terms.
  • BM25 can strongly reward precise identifiers, names, and uncommon query terms.
  • Raw score thresholds should be evaluated separately for each retrieval method.

Fuse ranked lists with RRF

RRF works from positions rather than raw scores. Retrieve a bounded list from dense search and another from BM25, then assign each document a fusion contribution based on its rank in each list. Documents appearing near the top of either list receive more credit, while documents present in both lists accumulate credit.

For a document d, a common formula is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of d in retrieval list i and k is a positive constant. The same document ID must be used to join results across both paths.

  • Request a candidate depth from both S3 Vectors dense search and Quickwit BM25 search.
  • Deduplicate candidates by a stable document or chunk identifier.
  • Sum the RRF contribution for every list in which a candidate appears.
  • Sort by the fused score and return the top requested results.

Make fusion observable before tuning it

Start with equal treatment for the dense and sparse lists, then inspect representative queries. Queries involving product codes, quoted phrases, or exact terminology may depend heavily on BM25. Broader natural-language questions may benefit from dense retrieval. The purpose of review is not to declare a permanent winner, but to understand which retrieval path contributes useful candidates.

Record provenance with each fused result: its dense rank, BM25 rank, and final fused rank. This makes debugging concrete. If an expected document disappears, teams can determine whether it was absent from both candidate lists, present but outranked, or excluded because candidate depth was too small.

  • Keep a small evaluation set containing exact-match and semantic queries.
  • Log document IDs and per-retriever ranks alongside fused results.
  • Review candidate depth before changing the fusion formula.
  • Treat changes to chunking, embeddings, and BM25 indexing as retrieval changes that merit re-evaluation.