Why raw-score fusion is fragile

Dense and sparse rankers emit scores with different semantics. A dense score reflects the relationship between embeddings under a chosen similarity function. A BM25 score reflects term statistics, document length normalization, and query-term contributions. Their numeric ranges can shift as models, indexes, corpora, and query shapes change.

Adding or averaging those scores therefore requires calibration that is easy to get wrong. A coefficient that appears reasonable for short keyword queries may behave differently for natural-language questions, rare identifiers, or recently indexed content. Rank-based fusion avoids requiring the two score distributions to be comparable.

RRF only asks a narrower question: did a document appear near the top of either ranked list? That makes it a useful baseline before investing in learned rerankers or score-normalization pipelines.

  • Use dense retrieval for semantic paraphrases and concept matches.
  • Use BM25 for exact names, codes, error messages, and uncommon terms.
  • Do not assume a score of 0.8 from one retriever is equivalent to 0.8 from another.
  • Preserve each retriever's ordered results and stable document identifiers.

Fuse two ranked lists with RRF

For each candidate document d, calculate an RRF score by summing 1 divided by k plus the document's rank from every list in which it appears: RRF(d) = Σ 1 / (k + rank_i(d)). Ranks are normally one-based, so the first result has rank 1. Documents absent from a list contribute nothing for that list.

The constant k dampens the effect of rank differences. A larger k makes the gap between adjacent positions less dramatic; a smaller k emphasizes the very top of each list. Start with a documented default such as 60, then evaluate it against representative queries rather than treating it as a universal optimum.

Retrieve a bounded candidate set from S3 Vectors and from Quickwit BM25, join the lists by canonical document ID, compute the fused score, then sort descending. Fetch result metadata after fusion when possible so the final presentation uses one consistent document record.

  • Request the same candidate depth from both retrievers initially, such as the top N results.
  • Deduplicate by a canonical ID, not by title or text equality.
  • Assign a deterministic secondary sort key for tied fused scores.
  • Log source ranks alongside the fused rank for debugging and relevance review.

Evaluate the fusion path as a retrieval system

RRF is not a substitute for evaluation. Build a small query set that includes semantic questions, exact-lookups, mixed queries, and cases with filters or access constraints. For each query, record whether the desired document appears in the review depth that matters to the application.

Inspect failure modes by source. If BM25 finds a document that dense retrieval misses, the issue may be embedding coverage, chunking, or vocabulary-specific content. If dense retrieval finds it but BM25 does not, the query may be phrased differently from the source text. Fusion visibility makes those differences actionable.

Keep retrieval and authorization separate in the design: apply required tenant, document-state, and access filters consistently to both candidate lists before fusion. A strong fused rank is irrelevant if the candidate was not eligible to be returned.

  • Compare dense-only, BM25-only, and RRF results on the same judged queries.
  • Review the top results, not only whether a relevant item appears somewhere in the candidate set.
  • Version query preprocessing, embedding models, and fusion parameters in evaluation records.
  • Re-run the query set after corpus, indexing, or ranking changes.