Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different ranking systems. Their ranges, distributions, and meanings can vary with the embedding model, query wording, corpus composition, tokenization, and index configuration. A weighted sum such as dense_score + bm25_score may appear straightforward, but its behavior can shift unexpectedly as any of those inputs change.

This is particularly risky when an exact-match query includes a rare identifier. BM25 may assign that document a very strong sparse score, while a semantic query may create a different dense-score distribution altogether. Without deliberate calibration and ongoing evaluation, score arithmetic can cause one retriever to dominate for reasons unrelated to relevance.

  • Dense scores reflect proximity in an embedding space.
  • BM25 scores reflect term-based matching and weighting.
  • Score magnitudes are not automatically comparable across retrievers.
  • Rank positions are often a more stable signal to combine.

Apply reciprocal rank fusion to two candidate lists

Reciprocal rank fusion (RRF) combines rankings rather than raw scores. Run dense retrieval against S3 Vectors and sparse retrieval with Quickwit BM25, request a candidate list from each, and assign every document a fused score based on its position in each list. A document appearing in both lists receives contributions from both rankings.

For each document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of d in ranking i and k is a positive constant. A commonly used starting value for k is 60, but it is a tuning choice rather than a universal rule. Sort documents by the resulting fused score and return the highest-ranked results.

  • Use one-based ranks: the first result has rank 1.
  • Assign no contribution when a document is absent from a list.
  • Deduplicate by a stable document identifier before returning results.
  • Preserve retriever-specific scores and ranks in logs for diagnosis.

Choose candidate depth and evaluate query slices

RRF can only promote documents that appear in at least one candidate list. If each retriever returns too few candidates, a useful document may never reach fusion. Start with a candidate depth larger than the final result count, then measure whether increasing that depth changes the quality or composition of the fused top results.

Evaluation should include query slices, not only an aggregate relevance number. Build examples for exact identifiers, short keyword queries, natural-language questions, terminology mismatch, and documents containing both specialized vocabulary and broad concepts. These slices reveal whether dense retrieval, BM25, or fusion is doing useful work for the queries your application actually receives.

  • Set a final result count separately from per-retriever candidate depth.
  • Test documents returned by both systems and documents unique to each system.
  • Inspect failures where a relevant result is missing from both candidate lists.
  • Version fusion settings alongside embedding and indexing changes.