Why raw-score merging is risky

A dense search score and a BM25 score are produced by different retrieval models. Their ranges, distributions, and meanings can differ even when both searches are run against the same document collection. Sorting a combined list by raw score can therefore cause one retrieval method to dominate simply because its numeric scale is larger.

This problem becomes more pronounced as content changes. Adding documents, changing chunk sizes, or revising text normalization can shift score distributions. A merge strategy based on a fixed score threshold may work on one corpus snapshot and become unstable later.

  • Dense retrieval ranks documents by vector similarity.
  • BM25 ranks documents using term-frequency and corpus statistics.
  • Neither score type should be presumed comparable without calibration.
  • Rank positions are easier to combine across retrieval methods.

Fuse ranked lists with RRF

Reciprocal Rank Fusion assigns each document a contribution based on its position in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus its rank for every list in which it appears. The constant k reduces the difference between nearby top ranks and is commonly selected as a tuning parameter rather than treated as a universal value.

For example, retrieve a bounded candidate list from dense search backed by regional S3 Vectors and another from Quickwit BM25. Join results by a stable document or chunk identifier, add each rank contribution, then sort descending by the fused score. A result appearing near the top of both lists rises naturally, while a strong result from only one method can still remain competitive.

  • Use one stable ID for the same chunk across dense and sparse indexes.
  • Keep ranks one-based: the first result has rank 1.
  • Retrieve enough candidates from each path to allow useful overlap.
  • Store source ranks during debugging so fused ordering is explainable.

Make fusion observable and testable

Treat fusion as part of the retrieval contract, not as an opaque post-processing step. Log the query, the candidate IDs returned by each retrieval path, their individual ranks, and the final fused order. This record makes it possible to investigate why an exact identifier query preferred BM25 or why a conceptual query was surfaced by dense search.

Evaluate changes with a small, representative query set before altering candidate depths or the RRF constant. Include exact names, error messages, abbreviations, natural-language questions, and queries that should return no useful result. The goal is not to make every query favor both methods; it is to ensure that combining them improves coverage without hiding important failures.

  • Track overlap between dense and BM25 candidate lists.
  • Inspect queries where the final top result came from only one path.
  • Version chunking, embedding, and text-processing changes.
  • Keep a no-result test set to detect overly broad retrieval.