Why raw-score blending is fragile

Dense retrieval ranks documents by vector similarity, while BM25 ranks them from term-frequency and document-statistics signals. Even when both systems return numeric scores, those numbers do not necessarily share a meaningful range, distribution, or interpretation.

For example, a document with a strong BM25 score is not automatically more relevant than a document with a high dense-similarity score. Adding the two values directly can make one retrieval path dominate merely because its score scale is larger. This can change unexpectedly as data, analyzers, embeddings, or query wording change.

  • Dense search helps retrieve semantically related wording.
  • BM25 helps retrieve exact terms, identifiers, and rare phrases.
  • Raw score magnitudes should not be assumed comparable across retrieval systems.

Fuse ranks with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each ranked list, rather than on its original retrieval score. For a document at rank r, its contribution is 1 divided by k plus r. The final RRF score is the sum of that contribution across the dense and sparse result lists.

A typical implementation retrieves the top N candidates from S3 Vectors and the top N candidates from Quickwit BM25, normalizes document identifiers, then merges duplicates by adding their RRF contributions. The constant k reduces the difference between nearby ranks and keeps the fusion focused on documents that appear near the top of either list.

  • RRF score: sum of 1 / (k + rank) across result lists.
  • Use the same canonical document ID in both indexes.
  • Choose a candidate depth N that provides enough material for fusion.
  • Start with one shared k value and evaluate changes against representative queries.

Make fusion observable and easy to revise

Store retrieval metadata alongside the fused result during testing: whether the document came from dense search, BM25, or both; its rank in each list; and its final fusion score. This makes it possible to explain why a result appeared and to identify queries where one path is consistently missing useful candidates.

RRF is a strong baseline because it has few assumptions, but it is not a substitute for evaluation. Build a small query set covering semantic questions, exact identifiers, abbreviations, and newly introduced terminology. Review the final ranked results, then adjust candidate depth, indexing choices, or downstream reranking only when the evidence supports it.

  • Log source ranks rather than relying only on the final ordering.
  • Test exact-match and semantic-query cases separately.
  • Watch for documents returned by both systems; agreement is often useful evidence.
  • Re-evaluate after changing embeddings, text processing, or corpus composition.