Why raw dense and BM25 scores should not be mixed casually

A dense retrieval score and a BM25 score are produced by different models and scoring systems. Their numeric ranges, distributions, and meanings can differ from one query to the next. Adding them together without calibration can make a result appear stronger simply because one scoring system emits larger values.

Rank-based fusion avoids that assumption. Instead of asking whether a dense score of one value equals a BM25 score of another, it asks a simpler question: how highly did each retrieval path rank the document? This makes the combination step easier to reason about and less dependent on score-scale details.

  • Use dense retrieval for semantic similarity.
  • Use BM25 for exact terms, product names, codes, and uncommon vocabulary.
  • Treat each engine's score as local to that engine unless it has been explicitly calibrated.
  • Retrieve a bounded candidate list from each path before fusion.

Fuse two ranked lists with RRF

For each document, RRF adds a contribution from every list in which the document appears. A common formulation is: RRF(document) = sum of 1 / (k + rank). Here, rank starts at 1 and k is a positive constant chosen by the application. Documents that rank well in both lists receive more support than documents found by only one path.

In a Talqora-oriented retrieval flow, an application can obtain dense candidates from regional S3 Vectors and sparse candidates from Quickwit BM25, then perform this small fusion step in its own retrieval service. The fused list can then be used directly or passed to a later application-specific stage.

  • Request the same candidate depth from both paths as a simple starting point.
  • Use a stable document identifier to join duplicate results across lists.
  • Assign each result its one-based rank within its own list.
  • Sum the RRF contributions and sort documents by the resulting value.

Make fusion observable and tune it with real queries

Hybrid retrieval is most useful when its behavior is visible. Log which source retrieved each final document, its dense rank, its BM25 rank, and its fused rank. These fields make it possible to investigate whether a result was supported by semantic similarity, lexical matching, or both.

Build a small evaluation set from representative queries, including acronym-heavy requests, exact identifier lookups, natural-language questions, and ambiguous wording. Review the top results before changing candidate depths or the RRF constant. The goal is not to force every query through the same behavior, but to preserve useful coverage across query types.

  • Track overlap between dense and BM25 candidate lists.
  • Inspect queries where one retrieval path contributes most of the final results.
  • Keep fusion configuration versioned with the application.
  • Test changes against a fixed query set before rollout.