Why rank fusion is safer than score fusion

Dense retrieval and BM25 produce scores with different meanings and ranges. A dense similarity score reflects the relationship between embeddings, while BM25 scores are derived from term-frequency and document-frequency signals. Adding those raw values together assumes they share a common scale, which is usually not a safe assumption.

RRF avoids that assumption. It uses only each result's position in a ranked list, then rewards documents that appear near the top of one or both lists. This makes it a useful baseline when combining regional S3 Vectors dense results with Quickwit BM25 results.

  • Run dense and sparse retrieval as separate queries.
  • Keep a stable document or chunk ID shared by both indexes.
  • Fuse ranks rather than directly comparing raw scores.
  • Return the highest fused results to the application or reranker.

Apply reciprocal rank fusion

For each candidate document d, calculate a fused score by summing 1 divided by k plus its rank for every list where it appears. In notation: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the difference between adjacent ranks and prevents the first few positions from dominating too sharply.

Choose the same retrieval depth for both sources at first, such as the top N dense results and top N BM25 results. Then inspect examples from real queries before changing depth, k, or adding source-specific weights.

  • Assign rank 1 to the first result in each list.
  • Use a consistent k value across evaluations; 60 is a commonly used starting convention, not a universal optimum.
  • Deduplicate by canonical chunk ID before returning results.
  • Use a deterministic tie-breaker, such as document ID, for stable output.

Evaluate the merged list with query slices

A single aggregate relevance number can hide where hybrid retrieval helps. Separate an evaluation set into query slices: exact identifiers, abbreviations, natural-language questions, recent terminology, and queries containing several constraints. Compare dense-only, BM25-only, and fused lists for each slice.

Review failures at the passage level, not only at the document level. If a useful document is retrieved but the wrong chunk is returned, the issue may be chunking or metadata design rather than the fusion rule.

  • Record which source retrieved each final candidate.
  • Measure recall at a fixed cutoff before optimizing ranking details.
  • Keep test queries separate from queries used to tune parameters.
  • Log index version and embedding version alongside evaluation results.