Why raw-score addition is fragile

A dense-search score and a BM25 score are produced by different models and ranking functions. Their ranges, distributions, and sensitivity to query wording can differ substantially. Adding them directly creates an implicit assumption: a one-point change in one system has the same meaning as a one-point change in the other.

That assumption may hold only after deliberate calibration, and calibration can drift as embeddings, analyzers, document fields, or corpora change. A query containing a model number may benefit heavily from sparse matching, while a natural-language question may benefit from semantic retrieval. Fixed score weights can make either behavior dominate unexpectedly.

  • Do not compare score magnitudes from separate retrieval systems without validating their meaning.
  • Avoid normalizing against only the current result set when stable behavior across queries matters.
  • Keep each retriever responsible for producing its own ranked candidate list.
  • Evaluate fused rankings with representative queries, including exact-match and semantic-paraphrase cases.

Fuse ranks, not scores, with RRF

Reciprocal rank fusion ignores the numerical scores and instead rewards documents for appearing near the top of each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank in every list where it appears. Documents absent from a list contribute nothing from that retriever.

The constant k reduces the gap between adjacent top ranks. A larger k makes the fusion less sensitive to small rank changes near rank one; a smaller k makes top positions more influential. Treat k as a relevance-tuning parameter, not as a universal default. The important property is that RRF requires only ordered lists, so it does not require dense and BM25 scores to share a scale.

  • Retrieve a bounded candidate list from S3 Vectors for the dense query.
  • Retrieve a bounded candidate list from Quickwit BM25 for the sparse query.
  • Assign ranks starting at 1 within each list.
  • For each document, sum 1 / (k + rank) across the lists where it appears, then sort descending.

Implement the fusion boundary deliberately

Use a stable document identifier in both retrieval paths. The fusion service or application should deduplicate by that identifier, retain source ranks for debugging, and apply access-control or tenant filters consistently before results reach the user. If filtering happens after a small candidate list is retrieved, relevant authorized documents may never enter the fusion pool.

Start with equal treatment of the dense and sparse lists. If evaluation shows one retriever should contribute more for a known query class, use a weighted RRF variant: multiply that retriever’s reciprocal-rank contribution by a documented weight. Keep query classification simple and observable; an opaque routing rule can make relevance regressions difficult to diagnose.

  • Log query type, candidate counts, ranks, fused score, and which retrievers returned each result.
  • Choose candidate depths large enough to create useful overlap and complementary candidates.
  • Test identifier-heavy, acronym-heavy, natural-language, and multi-concept queries separately.
  • Revisit k, candidate depths, and any weights when the corpus or retrieval configuration changes.