Why raw-score fusion is fragile

A dense search score and a BM25 score are not automatically measurements on the same scale. Their ranges, distributions, and meaning depend on the embedding model, distance or similarity choice, document collection, analyzer configuration, and query terms. A score of 0.8 from one retriever does not inherently mean the same thing as a score of 0.8 from another.

Min-max normalization can make scores look comparable, but it can be unstable for individual queries. One unusually strong result, a small candidate set, or a query with weak lexical matches may shift the normalized values substantially. Weighted score addition also requires ongoing tuning as content and retrieval configuration evolve.

  • Dense retrieval is often helpful when the query and document use different but related language.
  • BM25 is often helpful when exact identifiers, names, error codes, or uncommon terms matter.
  • Raw scores should be treated as retriever-specific signals unless they have been deliberately calibrated.

Fuse ranked lists with RRF

RRF starts with two independently ranked candidate lists: one from dense retrieval and one from BM25. For each document, add a contribution based on its rank in every list where it appears. The common formula is RRF(d) = Σ 1 / (k + rank_i(d)), where rank 1 is the top result and k is a damping constant.

Because RRF uses ordinal position rather than score magnitude, it avoids pretending that the two backends expose equivalent relevance scales. A document that ranks well in both lists rises naturally, while a document found by only one retriever can still be retained for recall.

  • Retrieve a candidate window from each retriever before fusion, rather than fusing only each system's final few results.
  • Use a consistent document identifier so results from S3 Vectors and Quickwit can be joined reliably.
  • Choose one rank convention and use it everywhere; RRF formulas usually treat the best item as rank 1.
  • Start with a conventional damping value such as 60, then validate it against representative queries.

Build an observable fusion path

A practical request path is simple: issue the dense and BM25 searches, collect their ranked document IDs, compute one RRF score per unique ID, sort by that fused score, and fetch the metadata needed for the response. Keep the original rank and source membership alongside the fused result during development. Those fields make it possible to explain why a result appeared.

Evaluation should include queries that separate retrieval modes. For example, test paraphrased questions, exact product names, part numbers, quoted phrases, and short ambiguous searches. Review not only aggregate relevance judgments but also whether the candidate window is large enough for either retriever to contribute useful documents.

  • Log dense rank, BM25 rank, fused rank, and whether a document was present in one or both lists.
  • Deduplicate by canonical document ID before sorting fused results.
  • Inspect zero-result and low-overlap queries; they often reveal analyzer, chunking, or embedding issues.
  • Treat fusion settings as versioned retrieval configuration so relevance changes can be traced.