Why raw-score merging is fragile

Dense retrieval scores describe a vector-space relationship between a query embedding and document embedding. BM25 scores are produced from lexical term matching, term frequency, document frequency, and document-length normalization. Even when higher is better in both systems, the numbers do not have a shared meaning.

Adding, averaging, or applying a fixed threshold to raw dense and sparse scores can make ranking sensitive to corpus changes, analyzer choices, embedding models, or query wording. A score that looks large in one result set is not automatically stronger evidence than a score from the other.

  • Dense search can surface paraphrases and conceptually related passages.
  • BM25 is especially useful for exact names, identifiers, error codes, and uncommon terms.
  • Raw score ranges may vary by query and by retrieval system.

Fuse result ranks with RRF

Reciprocal Rank Fusion (RRF) avoids direct comparison of scores. Retrieve a ranked candidate list from S3 Vectors and another from Quickwit BM25, then assign each document a contribution based on its rank in each list. Documents appearing near the top of either list receive more credit; documents appearing in both receive combined credit.

A common form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in result list i and k is a positive constant. The constant softens the difference between adjacent ranks, helping prevent a single first-place result from overwhelming the rest of the merged list.

  • Use stable document IDs shared by the dense and sparse indexes.
  • Retrieve a bounded candidate set from each system before fusion.
  • Deduplicate by document ID, summing contributions when an item appears in both lists.
  • Choose k deliberately and keep it configurable for offline evaluation.

Implement the merge as a small, observable retrieval stage

For each query, issue dense and sparse retrieval independently, retaining rank position and source metadata for every candidate. Build a map keyed by document ID, add the RRF contribution from each source, then sort by the fused score. Return the fused rank along with the component ranks so ranking behavior can be inspected later.

Keep candidate generation and fusion separate from later stages such as filtering, permissions checks, or reranking. This makes it easier to determine whether a poor result was absent from dense retrieval, absent from BM25, removed by filtering, or simply ranked too low after fusion.

  • Log query ID, candidate ID, dense rank, BM25 rank, and fused rank.
  • Measure overlap between dense and sparse candidate lists for representative queries.
  • Test exact-match-heavy and semantic-paraphrase-heavy query sets separately.
  • Use the same filters and access-control rules for both candidate sources.