Why raw dense and BM25 scores should not be added

A dense-search score expresses similarity in an embedding space. Its range and distribution depend on the embedding model, distance metric, index configuration, and even the query itself. A BM25 score instead reflects term frequency, inverse document frequency, and document-length normalization.

Because those scoring systems represent different signals, a formula such as dense_score + bm25_score can quietly make one retriever dominate. Min-max normalization may appear to fix the issue, but it is sensitive to the particular candidate set returned for each query and can behave unpredictably for short or highly specific queries.

  • Dense retrieval helps with paraphrases and conceptual similarity.
  • BM25 preserves exact-match signals for names, codes, error messages, and rare terms.
  • Raw score values are retrieval-system-specific, not a shared relevance currency.

Fuse ranks, not scores, with RRF

Reciprocal rank fusion assigns each document a contribution based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank in every list where it appears. The constant k reduces the advantage of being placed first and makes the method less sensitive to small rank changes.

For example, retrieve a top-N dense list from regional S3 Vectors and a top-N sparse list from Quickwit BM25. Deduplicate by a stable document or chunk identifier, calculate the RRF score, sort descending, and send the leading results to the next stage of the application.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first result has rank 1.
  • Start with the same candidate depth for both lists, then evaluate query logs before changing it.
  • Keep the source ranks in response metadata to make ranking behavior debuggable.

Make fusion operationally useful

RRF is most effective when both indexes refer to the same retrieval unit. If the dense path indexes chunks but the sparse path indexes whole documents, a fusion key can collapse distinct evidence or create duplicate-looking results. Choose and consistently store a canonical identifier for the unit users should receive: a document, passage, product record, or support article section.

Evaluate fused output using representative queries rather than only aggregate relevance judgments. Include exact identifier lookups, acronym-heavy questions, paraphrases, multi-concept queries, and queries with vocabulary absent from the source text. These categories reveal whether dense search and BM25 are contributing complementary results.

  • Apply identical tenant, permission, language, and freshness filters before fusion.
  • Deduplicate on a canonical retrieval-unit ID, not display text.
  • Log each candidate’s dense rank, BM25 rank, fused rank, and selected result.
  • If ranking quality needs further improvement, add a separate reranking step after fusion rather than forcing incompatible first-stage scores together.