Why score-based merging is fragile

A tempting approach is to take a dense similarity score and a BM25 score, normalize both, then add them with weights. This can work in a narrowly controlled evaluation set, but it is easy to make unstable. The score ranges, distributions, and meanings are different: a dense score reflects geometric similarity in an embedding space, while BM25 reflects term statistics and document-length normalization.

Those distributions can also shift as the corpus changes. Adding a large batch of documents, changing chunk sizes, updating an embedding model, or adjusting BM25 analysis can change score behavior. A fixed weighted formula may therefore need more monitoring and tuning than the retrieval problem warrants.

  • Do not assume a dense score of 0.8 has a comparable meaning to a BM25 score of 0.8.
  • Avoid selecting weights solely from a handful of representative-looking queries.
  • Treat corpus and indexing changes as potential score-distribution changes.
  • Prefer rank-based fusion when calibrated relevance scores are not available.

Fuse candidate lists with reciprocal rank fusion

Reciprocal rank fusion, usually abbreviated RRF, combines ranked lists rather than raw scores. For each document, add 1 divided by k plus its rank for every list in which it appears. The constant k reduces the influence of a single first-place result and keeps contributions from lower-ranked candidates useful.

For example, retrieve the top N documents from S3 Vectors and the top N from Quickwit BM25. Assign each document an RRF score, sort descending, then send the fused top results to the next stage of the application. Documents returned by both systems receive contributions from both rankings, while a strong result from only one system can still appear.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use the same stable document or chunk identifier in both retrieval paths.
  • Choose a candidate depth N that gives the fusion stage enough overlap and diversity.
  • Keep dense and sparse retrieval independent; fusion belongs after both ranked lists are available.

Make fusion observable before making it clever

Log the source ranks that produced each final result. A record such as dense rank 2, BM25 rank 18 is more actionable than a final fused score alone. It lets engineers see whether retrieval is driven by semantic similarity, lexical matching, or agreement between the two.

Evaluate with queries that reflect real failure modes: product names, acronyms, copied error strings, conversational questions, and queries with multiple constraints. Review not only whether a relevant document appears, but also whether it appears early enough for the consuming workflow. If one retrieval path repeatedly contributes no useful candidates for an important query class, investigate its indexing, chunking, metadata filters, or query construction before changing the fusion formula.

  • Log query text, applied filters, candidate IDs, source ranks, and final rank.
  • Measure result quality separately for exact-term and semantic-paraphrase query sets.
  • Inspect duplicate or near-duplicate chunks after fusion.
  • Version retrieval settings so evaluation results can be tied to a specific configuration.