Why raw-score blending is fragile

Dense retrieval and BM25 produce scores with different meanings and distributions. A vector similarity score reflects the relationship between embeddings, while a BM25 score depends on term frequency, document frequency, and document length. Adding these values together assumes that a change in one score has the same significance as a change in the other, which is rarely a safe assumption.

Score ranges can also shift across queries. A query containing a rare product code may create a sharply separated BM25 result set, whereas a broad natural-language question may yield more tightly grouped vector results. A fixed normalization rule can therefore behave differently from one query to the next.

  • Do not assume dense and BM25 scores share a common scale.
  • Avoid choosing a global weight solely from a small set of sample queries.
  • Preserve each retriever’s ranking before attempting to combine results.

Fuse ranks with a small, inspectable formula

RRF assigns each document a contribution based on its position in each result list rather than on its raw retrieval score. For a document d, calculate RRF(d) as the sum of 1 divided by k plus its rank in every list where it appears. The constant k reduces the difference between adjacent top positions and prevents a single rank-one result from overwhelming all other evidence.

For example, retrieve a candidate list from regional S3 Vectors and another from Quickwit BM25. Deduplicate documents by a stable document identifier, add their rank contributions, then sort descending by the fused value. A document that ranks well in both lists rises naturally, while a document that is uniquely strong in one list can still remain competitive.

  • Use one stable identifier when deduplicating dense and sparse candidates.
  • Treat rank 1 as the first result in each retrieval list.
  • Start with the same candidate depth for both retrievers, then inspect missed relevant results.
  • Keep k configurable so its effect can be evaluated against representative queries.

Make hybrid retrieval observable

A fusion rule is easier to improve when the application records how each final result entered the ranking. Log the dense rank, sparse rank, fused score, query type, and the selected document identifier. This makes it possible to distinguish results supported by both retrieval methods from results contributed by only one.

Review queries that represent real retrieval conditions: acronym-heavy requests, identifier lookups, paraphrased questions, and mixed natural-language-plus-keyword searches. If exact-match queries consistently need sparse retrieval to succeed, or paraphrases consistently depend on dense retrieval, that is useful evidence for candidate-depth and filtering decisions rather than a reason to force raw scores into one scale.

  • Inspect overlap between dense and sparse candidate sets.
  • Track whether relevant documents appear in either candidate list before fusion.
  • Evaluate rank changes, not only whether a document was retrieved.
  • Apply the same metadata filters to both paths when the use case requires consistent scope.