Why raw dense and BM25 scores should not be blended casually

A dense-search score and a BM25 score are produced by different retrieval models and have different ranges, distributions, and meanings. A score of 0.8 from one system is not inherently comparable to a score of 0.8 from another. Adding such values directly can make the result depend more on score scale than on document relevance.

Rank-based fusion avoids that comparison. Instead of treating the scores as interchangeable, it uses each retriever's ordering: a document near the top of either list receives more credit than one appearing farther down. This makes a useful baseline when relevance labels are limited or score calibration has not been established.

  • Dense retrieval helps with semantic similarity and paraphrased language.
  • BM25 helps preserve exact-term, rare-token, and identifier matches.
  • Rank positions are easier to combine than unrelated score scales.

Fuse candidate lists with reciprocal rank fusion

Run the same user query through dense retrieval and sparse BM25 retrieval, then retain a candidate list from each. For every document in the union of both lists, calculate an RRF score by summing 1 divided by k plus the document's rank for every list in which it appears. The constant k reduces the influence of small rank differences near the top.

For example, with k set to 60, a document ranked 2nd by dense retrieval and 9th by BM25 receives 1/(60+2) + 1/(60+9). A document returned by only one retriever can still rank well, but agreement between the lists is rewarded. Sort the merged candidates by this combined value before returning results or passing them to a later reranking step.

  • Retrieve a bounded candidate set from S3 Vectors and from Quickwit BM25.
  • Use one-based ranks: rank 1 is the first item in a result list.
  • Deduplicate documents using a stable document ID before scoring.
  • Choose and record one k value so experiments remain comparable.

Make hybrid retrieval observable and testable

Hybrid retrieval is a retrieval policy, not a guarantee that every query needs equal dense and sparse influence. Log which source retrieved each final document, its rank in each source list, and its fused rank. These fields make it possible to identify patterns such as identifier-heavy queries relying mostly on BM25 or broad natural-language questions benefiting from dense retrieval.

Evaluate with a small, representative query set before changing the policy. Include semantic paraphrases, exact names, abbreviations, error messages, and queries that intentionally combine concepts with identifiers. Review not only whether a relevant item appears, but also whether it appears high enough for the intended user experience.

  • Keep query text, retrieval configuration, ranks, and document IDs in evaluation records.
  • Check for documents missing from one index or mapped to inconsistent IDs.
  • Compare dense-only, BM25-only, and fused rankings on the same query set.
  • Treat fusion parameters as versioned retrieval configuration.