Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different models and formulas. Dense similarity reflects the relationship between embeddings, while BM25 weights query-term frequency, document-term frequency, and corpus statistics. A value from one system is not automatically on the same scale as a value from the other.

Adding uncalibrated scores can make ranking brittle. A minor index change, a different embedding model, or a query with repeated terms may alter score distributions and shift results unexpectedly. Rank-based fusion avoids treating either score as a universal relevance measurement.

  • Dense retrieval is useful for semantic paraphrases and related concepts.
  • BM25 is useful for exact wording, codes, product names, and rare terms.
  • Score ranges can vary by query and by retrieval system.
  • A shared ranking method should rely on information both systems reliably provide: result order.

Fuse two result lists with Reciprocal Rank Fusion

Run the same user query through dense retrieval backed by S3 Vectors and sparse retrieval backed by Quickwit BM25. Request a sufficiently deep candidate list from each path, then identify documents using a stable shared document ID. RRF gives each document a contribution based on its position in each list.

For every returned document, compute RRF(d) = sum of 1 / (k + rank), once for each list in which document d appears. The rank starts at 1, and k is a positive constant that reduces the difference between adjacent positions. Sum the contributions, sort descending, and return the top fused candidates.

  • Use the same canonical document ID in the dense and sparse indexes.
  • Keep ranks one-based: the first result has rank 1.
  • Choose k as an explicit configuration value and validate it with relevance judgments.
  • A document returned by both paths receives two contributions; a strong one-path result can still appear.

Make fusion observable before tuning it

Start by logging the dense rank, sparse rank, and fused rank for every selected result. These fields explain whether a document won because both retrievers agreed or because one retriever found an otherwise missed candidate. They are also useful when investigating surprising search results.

Evaluate fusion with a small, representative query set before changing production defaults. Include semantic queries, exact identifier queries, multi-term questions, and queries with ambiguous language. The objective is not to make every query use both paths equally; it is to improve the final ordering for the query mix your users actually submit.

  • Inspect overlap between dense and BM25 candidate lists.
  • Record zero-result cases separately for each retrieval path.
  • Review queries where fused top results differ substantially from either original list.
  • Re-evaluate after changing embedding models, analyzers, fields, or document chunking.