Why raw dense and BM25 scores should not be added directly

A dense retriever and a BM25 retriever produce scores for different reasons. A dense score reflects the relationship between a query embedding and document embedding. BM25 is driven by token overlap, term frequency, document length, and corpus-level term statistics. Even when both systems rank relevant documents highly, the numeric ranges and distributions of their scores can differ substantially.

Adding those scores together assumes that a one-unit change in either score has equivalent meaning. That assumption is rarely stable across query types, embedding models, index settings, or corpus changes. A query containing an exact part number, for example, may benefit heavily from BM25, while a paraphrased question may rely more on dense retrieval. Rank fusion avoids requiring a universal score conversion.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact tokens, uncommon terms, and identifiers.
  • Raw score scales can change independently as either retrieval system evolves.

Retrieve in parallel and fuse by reciprocal rank

For each query, retrieve a bounded candidate list from both retrieval paths: dense search from regional S3 Vectors and sparse BM25 search from Quickwit. Preserve each document's rank within its own list. Then merge document identifiers across the two lists and calculate a reciprocal rank fusion, or RRF, score for every unique candidate.

A common RRF form is score(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in retrieval list i, and k is a positive constant selected by the application. Documents returned by both systems receive contributions from both lists. Documents found by only one system can still enter the fused results when they rank well in that system.

  • Run dense and BM25 retrieval concurrently for the same query.
  • Choose a retrieval depth per path before fusion, such as a fixed top-N candidate list.
  • Deduplicate by a stable document or chunk identifier.
  • Sort unique candidates by descending fused score before returning the final top results.

Make fusion observable before making it more complex

Log enough information to understand why a result appeared: its fused rank, dense rank when present, BM25 rank when present, and the retrieval path or paths that contributed it. These fields make it possible to inspect failures such as an exact-match result being displaced by a broad semantic match, or a useful paraphrase never entering the candidate pool.

Start with one fusion rule and evaluate it against representative queries from the application. Include queries with names, codes, quoted text, abbreviations, natural-language questions, and ambiguous terminology. If a later reranking stage is added, use fused retrieval as candidate generation and continue recording which candidates each retrieval path supplied.

  • Track overlap between dense and BM25 candidate lists.
  • Review zero-result and low-confidence query samples separately.
  • Keep document identifiers consistent across both indexes.
  • Change candidate depth or fusion parameters one at a time so evaluation remains interpretable.