Why raw dense and BM25 scores should not be mixed casually

Dense retrieval typically ranks documents by a vector-similarity measure, while BM25 ranks them from term-frequency and corpus-statistics signals. Even when both produce higher-is-better scores, the numeric ranges, distributions, and meanings are different.

Adding or averaging those raw scores can make one retriever dominate for reasons unrelated to relevance. Score behavior may also shift as embeddings, analyzers, document lengths, or the indexed corpus change. A rank-based method avoids requiring a stable shared score scale.

  • Dense retrieval can recover paraphrases and conceptually related text.
  • BM25 can strongly reward exact identifiers, names, codes, and rare terms.
  • A fused ranking should preserve useful evidence from both result lists.

Apply Reciprocal Rank Fusion to two candidate lists

Run the same query through dense search in regional S3 Vectors and sparse search in Quickwit BM25. Keep a candidate list from each retriever, then assign every document a fusion contribution based on its rank rather than its original retrieval score.

For a document d, RRF can be written as: score(d) = Σ 1 / (k + rank_i(d)). The sum is taken across the result lists in which d appears. The constant k reduces the difference between adjacent top ranks and is commonly treated as a tunable parameter.

  • Use one-based ranks: the first result has rank 1.
  • Add a contribution only when the document appears in that retriever’s candidate list.
  • Deduplicate by a stable document identifier before returning the fused ranking.
  • Choose candidate depths large enough to give both retrievers an opportunity to contribute.

Validate fusion with query groups, not a single aggregate number

Evaluate fused retrieval against representative query groups. Include exact-match queries such as product codes or quoted phrases, semantic questions expressed in varied language, short ambiguous queries, and longer natural-language requests. This helps reveal whether hybrid retrieval improves coverage without hiding regressions in an important query type.

Inspect the source of each top result during debugging: dense only, BM25 only, or both. Documents found by both retrievers often receive a useful boost under RRF, while a strong result unique to either method can still surface.

  • Record the dense rank, BM25 rank, and final fused rank for judged queries.
  • Review cases where one retriever returns no useful candidates.
  • Tune candidate depth and the RRF constant using relevance judgments.
  • Keep fusion logic deterministic so ranking changes are easier to investigate.