Why raw score mixing is fragile

It is tempting to add a dense-search score to a BM25 score and sort by the result. This usually requires careful calibration because the two scores come from different retrieval models and do not necessarily share a meaningful scale. A score of 0.7 from one system is not inherently comparable to a score of 7 from another.

Rank-based fusion avoids that assumption. Instead of asking whether two scores mean the same thing, it asks a simpler question: which documents appear near the top of one or more result lists? This makes the merger easier to reason about and less dependent on score-distribution changes.

  • Run dense and sparse retrieval independently for the same query.
  • Keep document identifiers stable across both indexes.
  • Request a candidate depth larger than the final number of results.
  • Treat each source's result order as the input to fusion.

Apply Reciprocal Rank Fusion

For each candidate document, RRF adds a contribution from every ranked list in which that document appears. The contribution is 1 divided by k plus the document rank. The constant k reduces the difference between adjacent top ranks and prevents a single first-place result from dominating the merged list.

For a document d, the fused score can be written as: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across the dense and BM25 lists where d appears. Documents absent from a list simply receive no contribution from that list.

  • Use one-based ranks: the first result has rank 1.
  • Choose k as a configuration value and evaluate it on representative queries.
  • Deduplicate candidates by document ID before returning results.
  • Sort by fused score, then use a deterministic tie-breaker such as document ID.

Build the retrieval path around observability

In Talqora, dense candidates can come from regional S3 Vectors and sparse candidates from Quickwit BM25. The application layer can perform the RRF merge after receiving both ranked lists. Keeping this step explicit is useful because it makes retrieval policy visible, versionable, and testable.

Log which source contributed to each final result, its rank in each list, and the resulting fused score. These fields help diagnose cases where exact-match content is missing, semantic matches are overly broad, or one retrieval path returns too few candidates.

  • Store the dense rank, BM25 rank, and fused score with query traces.
  • Create a small query set containing paraphrases, identifiers, and mixed natural-language questions.
  • Review candidate depth separately from the final result count.
  • Change one fusion parameter at a time and compare result quality qualitatively or with labeled relevance judgments.