Why raw scores should not be compared directly

A dense-search score and a BM25 score are produced by different ranking models. Their ranges, distributions, and meanings are not inherently aligned. A value that looks larger in one system is not necessarily stronger evidence of relevance than a smaller value in the other.

This makes a simple strategy such as sorting a merged result set by raw score unreliable. It can unintentionally favor whichever retriever emits numerically larger values, even when the other retriever found the more useful document.

  • BM25 scores depend on term frequency, document frequency, and field normalization.
  • Dense scores depend on the embedding representation and similarity calculation.
  • Score distributions can shift as documents, queries, or embedding models change.
  • A stable merge should avoid assuming score scales are interchangeable.

Fuse ranked lists with reciprocal rank fusion

Reciprocal rank fusion, or RRF, merges result lists using positions rather than raw scores. For every document returned by a retriever, assign a contribution based on its rank. Documents that appear near the top of one or both lists receive the strongest combined signal.

A common form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in list i and k is a smoothing constant. The constant reduces the difference between adjacent top positions and helps prevent a single first-place result from dominating the entire merge.

  • Query S3 Vectors with the embedded user query to obtain a dense candidate list.
  • Query Quickwit BM25 with the text query to obtain a sparse candidate list.
  • Deduplicate documents by a stable document identifier.
  • Add each document's reciprocal-rank contributions, then sort by the fused total.

Make fusion observable before making it complex

Start with a fixed candidate depth for each retriever and a single RRF constant. Log which retrieval path contributed to each final result, its original rank, and its fused score. This makes it possible to identify whether a result was supported by semantic similarity, lexical matching, or both.

Evaluate the merged results with queries drawn from real retrieval tasks. Include exact identifiers, product names, short keyword queries, long natural-language questions, and queries with synonyms. The goal is not to prove that one retriever wins universally; it is to find a combination that preserves useful exact matches while expanding semantic recall.

  • Keep dense and sparse candidate depths configurable.
  • Record per-retriever ranks alongside the final fused rank.
  • Inspect queries where only one retriever contributes useful results.
  • Change one fusion parameter at a time and compare judged results.