Why fuse ranks instead of raw scores?

A dense retriever and a BM25 retriever produce scores with different meanings. A similarity score from dense search is not automatically comparable to a BM25 relevance score, even when both are returned for the same query.

Trying to add or average those raw values can make the result dependent on score distributions, index settings, and query shape. Rank-based fusion avoids that comparison: it uses each retriever's ordering rather than assuming their score scales are compatible.

  • Dense retrieval helps when query and document use different wording.
  • BM25 helps when exact names, identifiers, and rare terms matter.
  • Rank fusion combines evidence without requiring score calibration.

Apply Reciprocal Rank Fusion

With RRF, run a dense search and a sparse BM25 search independently, retaining a sufficiently deep candidate list from each. For every document that appears in either list, add a contribution based on its rank in that list.

The standard form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in result list i. The constant k reduces the advantage of a document appearing at the very top of a single list and is commonly treated as a tunable policy choice.

  • Use one-based ranks: first place has rank 1.
  • Deduplicate documents by a stable document ID before sorting the fused output.
  • Keep the per-retriever ranks for debugging and relevance review.
  • Retrieve more candidates than the final number of results you plan to return.

Make fusion observable and query-aware

In a Talqora-oriented architecture, dense candidates can come from regional S3 Vectors and sparse candidates from Quickwit BM25. The fusion step can live in the application layer, where it can preserve source metadata and return a single ranked response to the caller.

Start with a single, documented RRF configuration and evaluate it using representative queries. Include queries with product names, error messages, abbreviations, paraphrases, and multi-concept questions. The goal is not to prove that one retriever always wins, but to identify when their combination produces more useful candidates.

  • Log the dense rank, BM25 rank, and fused rank for returned documents.
  • Review queries where only one retrieval path contributed results.
  • Separate lexical-heavy queries from semantic paraphrase queries in evaluation.
  • Treat candidate depth and the RRF constant as relevance parameters, not fixed truths.