Why raw-score fusion is fragile

A dense retriever commonly returns a similarity or distance-derived score, while BM25 returns a lexical relevance score based on term frequency and corpus statistics. Even if both lists are sorted correctly on their own, a score of 0.8 from one system does not have a universal relationship to a score of 8 from the other.

Score normalization can work, but it adds assumptions. Min-max normalization depends on the candidate set, and distribution-based methods require representative query traffic. If rankings are the reliable output you already have, fuse the rankings directly instead of treating unlike scores as a shared measurement.

  • Use dense retrieval for semantic similarity and paraphrases.
  • Use BM25 for exact terms, identifiers, and uncommon vocabulary.
  • Do not assume dense and sparse scores can be added without calibration.
  • Keep each retriever’s candidate list and rank position available to the fusion layer.

Fuse candidate lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each ranked list: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken over every retriever list containing document d. A document appearing near the top of either list receives a meaningful contribution; a document supported by both lists receives two contributions.

The constant k reduces the difference between adjacent ranks and prevents the first few positions from dominating too sharply. A commonly used starting value is 60, but it is a tuning choice rather than a universal rule. Evaluate it with representative queries and relevance judgments when those are available.

  • Retrieve a bounded candidate set from S3 Vectors for the dense query.
  • Retrieve a bounded candidate set from Quickwit BM25 for the same user query.
  • Deduplicate results using a stable document or chunk identifier.
  • Sum reciprocal-rank contributions, then sort by the fused score.

Make fusion operationally useful

Use the same chunk identity across both indexes. If one index stores a source document ID while the other stores chunk IDs, fusion can accidentally combine mismatched units or show duplicate passages. Store metadata such as source ID, chunk ID, version, language, access scope, and deletion state consistently enough to filter both retrieval paths.

Log more than the final result list. Record whether each returned item came from dense search, BM25, or both, along with its rank in each list. These signals help diagnose failures: an exact-code query that only succeeds through BM25 and a paraphrased question that only succeeds through dense retrieval are expected patterns, not necessarily defects.

  • Apply authorization and tenant filters consistently before fusion.
  • Use a shared chunk ID to deduplicate dense and sparse candidates.
  • Inspect source attribution in logs for failed and successful queries.
  • Tune candidate depth and k together; shallow lists can hide useful overlap.