Why raw dense and BM25 scores are difficult to merge

A dense retriever ranks documents by a vector-similarity measure. A BM25 retriever ranks documents according to term frequency, document frequency, and query-term matching. Even when both systems return a numerical score, those values do not necessarily mean the same thing.

Adding or averaging uncalibrated scores can make the combined ranking sensitive to index settings, query length, embedding behavior, and score distributions. A safer default is to combine ordering information rather than assuming that a score of 0.8 from one retriever is comparable to 0.8 from another.

  • Dense search is useful when relevant wording differs from the query.
  • BM25 is useful for exact tokens, product names, codes, and rare terminology.
  • Score ranges can vary by query and retrieval method.
  • Rank positions are easier to interpret consistently than raw scores.

Fuse ranked lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each ranked result list. For a document d, the fused score is the sum of 1 divided by k plus its rank across all lists where it appears. The constant k reduces the impact of small rank differences near the top of a list.

For example, a document ranked third by dense search and tenth by BM25 receives contributions from both lists. A document found only by one retriever can still rank well, but agreement between retrievers is rewarded without requiring their raw scores to be normalized.

  • RRF formula: score(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: first place has rank 1.
  • Choose a fixed k and evaluate it against representative queries.
  • Deduplicate documents by a stable document identifier before returning results.

Apply the pattern in a Talqora retrieval pipeline

In a Talqora-oriented pipeline, issue the dense retrieval request against regional S3 Vectors and the sparse retrieval request against Quickwit BM25. Request a candidate set from each system, then perform RRF in the application or retrieval service that coordinates the two responses.

Keep the first implementation observable. Record which retrievers returned each final document, its rank in each list, and its fused rank. This makes it possible to investigate whether a missed result was absent from dense retrieval, absent from BM25, truncated by candidate depth, or displaced during fusion.

  • Retrieve more candidates than the final number of results you plan to show.
  • Use the same filters and tenant boundaries for both retrieval paths.
  • Store provenance such as dense rank, BM25 rank, and fused score.
  • Evaluate with real query-document relevance judgments before adding complexity.