Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different retrieval models and scales. Even when each result list is useful on its own, a score of 0.8 from one system has no inherent relationship to a score of 12 from the other.

Adding or averaging those values creates a hidden calibration problem. Changes to embeddings, document processing, BM25 configuration, or corpus composition can shift score distributions and unexpectedly change which results win.

  • Dense retrieval can surface semantically related wording that shares few exact terms.
  • BM25 can prioritize exact identifiers, product names, error messages, and rare terms.
  • Rank position is easier to compare across retrieval methods than raw score magnitude.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its position in each ranked list, then sums those contributions. For a document d, use RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i is its one-based position in retrieval list i and k is a positive constant.

Issue a dense query against the S3 Vectors-backed path and a sparse query against the Quickwit BM25-backed path. Merge results by a stable document ID, calculate the RRF total, and sort descending. A document returned by both methods receives two contributions; one returned by only one method can still rank well if it appears near the top.

  • Start with the same candidate depth for both lists, such as the top 50 or top 100 results.
  • Use a stable ID shared by dense and sparse indexes for deduplication.
  • Choose a fixed k initially and evaluate it against representative queries rather than interpreting it as a universal constant.
  • Keep the original rank and source list for each fused result to make debugging possible.

Make fusion operationally reliable

Apply the same tenant, authorization, language, and document-status constraints to both retrieval paths. Fusion cannot correct mismatched candidate sets: if one path can retrieve documents the other path was never allowed to see, the combined ranking may violate application expectations.

Evaluate the fused list with query groups that reveal complementary behavior. Include exact lookup queries, vague natural-language questions, synonym-heavy queries, and queries containing identifiers. Review not only relevance at the top of the list, but also whether either retrieval method consistently contributes useful unique documents.

  • Log query identifiers, per-path latency, candidate counts, ranks, and fused positions.
  • Define behavior for a partial result when one retrieval path is unavailable or returns no candidates.
  • Version embeddings and indexing configurations so ranking changes can be traced to a retrieval change.
  • Re-run an evaluation set after changing candidate depth, fusion parameters, analyzers, or embedding models.