Why raw dense and sparse scores should not be added directly

A dense retriever returns scores derived from an embedding similarity function. A BM25 retriever returns scores based on term frequency, document frequency, and document-length normalization. Even when both are useful signals, a score of 0.7 from one system has no inherent relationship to a score of 7 from the other.

Adding or weighting raw scores can therefore make ranking sensitive to query shape, corpus changes, analyzer choices, embedding models, and implementation details. A weight that appears reasonable for one query class may cause another retriever to dominate elsewhere. This is especially likely when queries range from natural-language questions to product names, error codes, or abbreviations.

  • Dense similarity and BM25 use different scoring models.
  • Score ranges can shift as indexes, models, and content change.
  • Per-query score normalization introduces its own assumptions.
  • Rank positions are usually easier to compare than raw values.

Fuse ranked lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in every result list where it appears. For a document d, calculate RRF(d) = Σ 1 / (k + rankᵢ(d)), summing across retrievers. The rank starts at 1, and k is a constant that reduces the influence of very high positions relative to the rest of the candidate list.

For a Talqora retrieval flow, run dense search against regional S3 Vectors and sparse search through Quickwit BM25. Take a bounded top-N list from each result, deduplicate documents by a stable identifier, compute the fused score, and sort descending. The resulting list rewards documents found by both methods while still preserving strong single-retriever matches.

  • Use a stable document or chunk ID for deduplication.
  • Request the same candidate depth from dense and sparse retrieval as a simple starting point.
  • Choose a fixed k and record it with retrieval configuration.
  • Keep each source rank for debugging and offline evaluation.

Tune the candidate set before tuning the formula

RRF is simple, but it still depends on the lists supplied to it. If the dense or BM25 candidate depth is too shallow, relevant documents cannot be recovered during fusion. Start by examining whether known relevant items appear in either candidate list before focusing on the final ordering.

Evaluation should include query classes that expose each retriever's strengths. Natural-language paraphrases can test dense recall, while exact names, codes, and quoted terms can test sparse recall. Review not only aggregate relevance metrics but also disagreement cases: documents returned by one retriever but not the other often reveal missing content, chunking problems, metadata filters, or analyzer issues.

  • Measure candidate recall separately for dense, sparse, and fused retrieval.
  • Inspect queries where one retriever finds relevant content and the other does not.
  • Apply identical authorization and metadata filters before presenting results.
  • Version embedding, chunking, and fusion settings so ranking changes are explainable.