Why raw-score blending is fragile

A dense retrieval score and a BM25 score are not automatically comparable. They may have different ranges, distributions, and meanings, even when both are returned as numbers. Adding them directly can make the result depend more on score calibration than on relevance.

This issue becomes especially visible when query types vary. A natural-language question may favor dense retrieval, while a query containing a product code, error string, or proper noun may depend heavily on sparse matching. A fixed weighted sum can behave inconsistently across those cases.

  • Dense scores reflect similarity in an embedding space.
  • BM25 ranks documents from term occurrences and term statistics.
  • Score ranges can change with indexing, query wording, or retrieval configuration.
  • Rank position is often safer to combine than raw score.

Apply Reciprocal Rank Fusion

RRF combines result lists using rank rather than the original retrieval scores. For each document, add a contribution from every list in which it appears: 1 divided by k plus its rank. The constant k reduces the effect of small rank differences near the top of a list.

For example, retrieve a candidate list from Talqora Vector's dense search path, backed by regional S3 Vectors, and another from its sparse Quickwit BM25 path. Assign ranks within each list, sum the RRF contributions by document ID, then sort the merged candidates by the resulting fused score.

  • Use one-based ranks: rank 1 is the first result.
  • A common formulation is RRF(d) = Σ 1 / (k + rank_i(d)).
  • Choose k as a tuning parameter and keep it consistent while evaluating changes.
  • Deduplicate by a stable document or chunk identifier before returning results.

Make fusion observable and testable

RRF is simple, but it should still be evaluated against representative queries. Build a small set that includes semantic questions, exact-name lookups, rare terminology, and mixed queries. Review not only whether a relevant item appears, but whether it reaches the portion of the ranking users actually inspect.

Log the dense rank, sparse rank, and fused rank for returned items. These fields make it possible to diagnose whether a result won because both retrievers agreed or because one retriever supplied a uniquely useful candidate. They also provide a clear basis for later tuning.

  • Record which retrieval paths returned each result.
  • Compare dense-only, sparse-only, and fused rankings on the same query set.
  • Inspect failures involving exact strings, abbreviations, and ambiguous language.
  • Treat candidate-list depth and k as evaluation-driven configuration choices.