Why rank fusion is safer than score fusion

Dense-search similarity scores and BM25 scores are not naturally comparable. Their ranges, distributions, and sensitivity to query length can differ substantially. Adding or averaging raw scores can therefore make one retrieval method dominate for reasons unrelated to relevance.

RRF avoids this calibration problem by using rank position rather than the original score. Talqora’s architecture—regional S3 Vectors for dense search and Quickwit BM25 for sparse search—maps naturally to this pattern: issue equivalent queries to both retrieval paths, then combine the returned rankings in an application-side fusion step.

  • Use dense retrieval for semantic similarity and paraphrased intent.
  • Use BM25 for exact terminology, product codes, names, and uncommon tokens.
  • Keep each retriever’s native scoring model intact.
  • Fuse only the ordered result lists.

Apply Reciprocal Rank Fusion

For each candidate document, RRF sums a small contribution from every result list in which that document appears. A common form is: RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in list i and k is a constant that reduces the impact of small rank differences.

Start with the same retrieval depth for both sources, such as the top 50 or top 100 results, and use a single k value consistently during evaluation. The important property is not a universally perfect constant; it is that a document appearing near the top of either list receives useful credit without requiring dense and sparse scores to share a scale.

  • Deduplicate candidates by a stable document or chunk identifier.
  • Treat rank 1 as the highest-ranked result.
  • Assign no contribution when a document is absent from a list.
  • Sort candidates by fused score and return the desired final top-k.

Evaluate the fusion pipeline, not just each retriever

Hybrid retrieval adds moving parts: query construction, retrieval depth, chunking, metadata filters, fusion, and any later reranking step. Evaluate the complete path against representative queries and judged relevant documents, rather than assuming that two individually useful retrievers always combine well.

Inspect failures by query type. If exact identifiers disappear, verify sparse recall and tokenization. If paraphrases fail, examine embedding choice, chunk boundaries, and dense retrieval depth. If results look repetitive, consider whether multiple chunks from one source document should be limited before presentation.

  • Create a test set containing both semantic and exact-match queries.
  • Measure recall at the candidate depth before evaluating final top-k ranking.
  • Log which retrieval path contributed each fused candidate.
  • Revisit fusion settings after changes to corpus content, chunking, or embeddings.