Why raw-score merging is fragile

Dense retrieval typically ranks records by a vector similarity or distance measure, while BM25 ranks records using term frequency, document frequency, and length normalization. Even when both systems return numeric scores, the magnitude and distribution of those scores represent different things.

A naive approach such as adding, averaging, or thresholding raw scores can make ranking sensitive to query wording, corpus changes, and implementation details. One retriever may dominate simply because its scores have a wider range, not because its results are more relevant.

Rank-based fusion takes a narrower and often more stable view: each retriever contributes evidence through the position of a result in its own ranked list.

  • Use dense search for semantic similarity and paraphrases.
  • Use BM25 for exact terms, identifiers, rare phrases, and lexical matches.
  • Avoid assuming a BM25 score and a dense similarity score share a common scale.

Fuse dense and sparse candidates with RRF

For a query, request a ranked candidate list from S3 Vectors and a ranked candidate list from Quickwit BM25. Then assign each document an RRF contribution from every list in which it appears. The contribution is 1 divided by k plus the document's one-based rank.

The fused score for a document is the sum of those contributions. A document appearing near the top of both lists rises naturally, while a strong result from only one retriever can still remain competitive. The constant k reduces the difference between adjacent ranks and prevents the first position from overwhelming the rest of the list.

For example, with k set to 60, a document at rank 2 in dense search and rank 8 in BM25 receives 1/(60+2) + 1/(60+8). The exact fused value matters less than applying the same formula consistently before sorting candidates.

  • Fetch enough candidates from each retriever to allow overlap and recovery of complementary results.
  • Deduplicate by a stable document or chunk identifier before calculating the final ordering.
  • Treat missing documents as contributing zero from the list where they do not appear.
  • Start with an established k value such as 60, then validate changes using judged queries.

Evaluate fusion with query classes, not intuition

RRF is simple, but it is still a retrieval policy that should be tested against the queries your application receives. Build a small evaluation set with expected relevant documents or passages, then compare dense-only, BM25-only, and fused rankings at the cutoff that matters to the next stage of your system.

Segment the evaluation set by query type. Product codes, error messages, names, and quoted text often test lexical retrieval. Natural-language questions, alternate phrasings, and conceptual descriptions test semantic retrieval. Hybrid fusion should be assessed on both groups rather than only on aggregate results.

If a downstream reranker or generative system consumes the retrieved items, inspect the candidate set as well as the final answer. The immediate goal of fusion is to place useful evidence in the available retrieval window.

  • Track relevance at a fixed cutoff such as the top 10 or top 20 results.
  • Keep failed queries and near-misses as permanent regression cases.
  • Log the source rank from dense search and BM25 for fused results.
  • Re-evaluate after changing chunking, embeddings, analyzers, or document fields.