Why raw-score fusion is fragile

Dense search and BM25 optimize different signals. Dense retrieval ranks documents by proximity between embeddings, while BM25 ranks them from lexical term matching and document statistics. Even when both systems return a numeric score, the values do not necessarily have the same range, distribution, or meaning.

Adding or averaging raw scores can therefore produce unstable results. A small change to an embedding model, an indexing configuration, a query’s wording, or the document corpus can shift one score distribution enough to dominate the combined ranking.

  • Dense retrieval can recover paraphrases and conceptually related passages.
  • BM25 can strongly reward exact identifiers, product names, error codes, and rare terms.
  • A score that is high within one retrieval system is not automatically high relative to another system.

Fuse ranks, not scores, with RRF

RRF replaces score comparison with rank comparison. Retrieve a candidate list from regional S3 Vectors and a candidate list from Quickwit BM25, then assign each document a contribution based on its position in each list. Documents found by both retrieval methods receive contributions from both lists.

For a document d, a common formulation is RRF(d) = sum of 1 / (k + rank_i(d)) across retrieval lists. The constant k reduces the advantage of appearing in the very first position and makes the fusion less sensitive to minor ordering differences near the top.

  • Run dense and sparse retrieval independently for the same query.
  • Use a stable document identifier to deduplicate results across both lists.
  • Assign ranks starting at 1 and sum the RRF contribution for every list containing the document.
  • Sort by the fused score, then return the desired number of results.

Choose candidate depth and evaluate query slices

RRF can only promote documents that enter at least one candidate set, so candidate depth matters. If each retriever returns too few documents, a document that is moderately ranked by both methods may never be available for fusion. Start with a candidate depth larger than the final number of results, then evaluate whether increasing it changes relevant outcomes.

Evaluate hybrid retrieval by query class rather than relying only on a single aggregate metric. Exact-match queries, natural-language questions, acronym-heavy queries, and multi-concept queries can reveal where dense and sparse retrieval complement each other. This makes it easier to decide whether to adjust candidate depth, the RRF constant, or the underlying retrieval inputs.

  • Keep the final result count separate from each retriever’s candidate count.
  • Log source ranks alongside the fused rank for debugging.
  • Inspect queries where only dense or only BM25 retrieved the relevant document.
  • Use a fixed evaluation set when changing fusion parameters.