Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score come from different retrieval models and scoring assumptions. Even if both are returned as numbers, their ranges, distributions, and meanings are not inherently aligned. A document with a strong BM25 score is not automatically more or less relevant than a document with a strong vector similarity score.

Adding raw scores can make ranking sensitive to implementation details rather than relevance. A change in BM25 configuration, query length, vector similarity metric, or corpus composition may alter score magnitudes without representing a comparable change in document quality.

  • BM25 rewards term-frequency and rarity signals.
  • Dense retrieval emphasizes semantic proximity in embedding space.
  • Score ranges can shift across queries and indexes.
  • A fusion method should avoid assuming one numeric scale.

Fuse ranks with Reciprocal Rank Fusion

Reciprocal Rank Fusion (RRF) combines ranked lists rather than their raw scores. Retrieve a candidate list from dense search and another from BM25, then assign each document a contribution based on its position in each list. Documents appearing near the top of either list receive more credit; documents appearing in both lists accumulate credit.

For a document d, calculate RRF(d) as the sum across retrieval lists of 1 divided by k plus rank(d). The constant k softens the difference between adjacent ranks. A commonly used starting point is 60, but it is a tuning parameter rather than a universal rule.

  • Request the same candidate depth from dense and sparse retrieval where practical.
  • Deduplicate documents using a stable document identifier.
  • Treat a missing document in a list as contributing zero for that list.
  • Sort the merged candidates by descending RRF score.

Implement a small, inspectable fusion layer

In an API-first retrieval design, the application can issue a dense request against regional S3 Vectors and a sparse BM25 request through Quickwit, then fuse the returned rankings in a small service or request handler. Keeping fusion logic explicit makes it easier to log inputs, inspect surprising results, and evolve ranking policy without conflating the two retrieval systems.

Start by preserving retrieval provenance for every fused result: its dense rank, BM25 rank, source scores, and final RRF score. This record is useful when evaluating queries such as product codes, error messages, acronyms, and paraphrased natural-language questions. Those query types often reveal whether one retrieval path is underrepresented in the candidate set.

  • Log ranks and source membership alongside the fused result.
  • Evaluate with representative queries, not only broad natural-language prompts.
  • Tune candidate depth and k together using relevance judgments.
  • Add later reranking only after the fused candidate set is reliable.