Why raw dense and BM25 scores should not be added blindly

A dense-search score and a BM25 score are produced by different retrieval models with different ranges and meanings. Even when both systems return a numeric relevance value, a score of 0.8 from one system is not inherently equivalent to 0.8 from the other.

Adding raw scores can cause one retriever to dominate solely because of its score distribution. That distribution may also change with embedding models, indexing settings, corpus composition, or BM25 configuration. A rank-based method avoids depending on cross-system score calibration.

  • Dense retrieval is useful when query wording differs from document wording.
  • BM25 is useful for exact terms, identifiers, names, and rare phrases.
  • Score scales are model-specific and should be treated as incomparable by default.

Fuse result lists with reciprocal rank fusion

Reciprocal rank fusion, commonly abbreviated as RRF, assigns each document a contribution based on its rank in each result list. For a document d, the fused score can be written as: RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the document rank from retriever i and k is a positive constant.

Run a dense query against regional S3 Vectors and a sparse query against Quickwit BM25, then take a candidate list from each. Deduplicate documents by a stable document ID, calculate their RRF scores, and sort the merged set by the fused score. A document appearing near the top of both lists receives a stronger combined signal than one appearing in only one list.

  • Use the same query text for both retrievers unless your application has a deliberate query-rewriting step.
  • Request a candidate depth larger than the final number of results shown to users.
  • Keep ranks one-based when implementing the RRF formula.
  • Choose and document a fixed k value before tuning it with relevance judgments.

Make hybrid retrieval observable and testable

Log the dense rank, sparse rank, and fused rank for returned documents. These fields make it possible to understand whether a result was supported by semantic similarity, lexical matching, or both. They also help diagnose regressions after changing an embedding model, document chunking strategy, or sparse index configuration.

Evaluate the fused ranking on a representative query set rather than relying on a few memorable examples. Include queries with exact product terms, error messages, paraphrased questions, and multi-concept requests. The goal is not for dense and sparse retrieval to agree on every query; the goal is for their differences to provide useful coverage.

  • Record document IDs and ranks from each retriever before fusion.
  • Track queries where only one retriever contributed a final result.
  • Use stable relevance labels when comparing changes over time.
  • Inspect duplicate and near-duplicate chunks, which can occupy multiple top ranks.