Why raw dense and BM25 scores should not be added casually

Dense retrieval and BM25 produce scores for different reasons. A dense score reflects proximity in an embedding space, while BM25 reflects term-based relevance using document and query statistics. Even when both searches return results for the same query, a score of 0.8 from one system has no inherent relationship to a score of 0.8 from the other.

Adding or averaging raw scores can make a retrieval pipeline sensitive to implementation details such as embedding choice, index configuration, document length, or score normalization. The resulting ranking may look stable for one query set and shift unexpectedly when either retrieval path changes.

Rank-based fusion avoids this calibration problem. It only asks where a document appeared in each ranked list, which is information every retrieval API can provide.

  • Dense retrieval can surface semantic matches that do not share query terms.
  • BM25 can favor exact names, identifiers, error codes, and rare terminology.
  • Raw scores from separate retrieval methods are not automatically comparable.

Apply reciprocal rank fusion to the two result lists

Run the dense query against the S3 Vectors-backed path and the lexical query against the Quickwit BM25 path. Request a sufficiently deep candidate list from each path, then use a stable document identifier to combine duplicate results.

For each document, add a contribution from every list in which it appears: 1 divided by k plus its one-based rank. The constant k reduces the difference between documents near the top of a list and makes fusion less dominated by a single rank position. A commonly used starting value is 60, but it is a tuning choice rather than a universal rule.

Sort documents by their summed RRF score, then return the top results. A document found by both retrieval methods is rewarded, while a strong result from only one method can still remain competitive.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first result has rank 1.
  • Deduplicate by a canonical document or chunk ID before returning results.
  • Keep the source ranks in logs for later relevance analysis.

Make fusion observable before making it more complex

Start with equal treatment for the dense and BM25 lists. This produces a clear baseline and makes it easier to understand whether each retrieval method contributes useful candidates. If results are later judged with query logs or relevance labels, weights, candidate depths, and k can be adjusted deliberately.

Inspect not only the final ranking but also the overlap between lists. Queries with little overlap are often informative: they may contain specialized vocabulary that favors BM25, broader concepts that favor dense retrieval, or corpus fields that need better indexing and chunking.

The API-first boundary is useful here. The application can keep retrieval, fusion, logging, and reranking as explicit stages, rather than hiding ranking decisions inside a single opaque score.

  • Record query text, candidate IDs, ranks, and the final fused order.
  • Compare dense-only, BM25-only, and fused results during evaluation.
  • Use the same filtering rules for both candidate lists when possible.
  • Add reranking only after the retrieval baseline is understood.