Why raw dense and sparse scores should not be added

Dense retrieval ranks documents by the relationship between vector representations of a query and document. BM25 ranks documents from lexical evidence such as term occurrence, term frequency, and document-length effects. Even when both systems return a numeric score, the magnitude and distribution of those numbers have different meanings.

Adding a vector similarity value to a BM25 score assumes that one point from each system represents comparable relevance. That assumption can change across indexes, queries, embedding models, analyzers, and corpus updates. A fusion method based on rank avoids depending on that assumption.

  • Dense retrieval can surface paraphrases and conceptually related text.
  • BM25 can reward exact terms, identifiers, error codes, and uncommon names.
  • Score ranges may vary by query, so fixed weighting of raw scores is difficult to reason about.
  • Rank positions are directly available from both result lists.

Fuse the two ranked lists with RRF

Run the same user query through dense retrieval in S3 Vectors and sparse BM25 retrieval in Quickwit. Keep a stable document identifier with every result. For each unique document returned by either system, calculate an RRF score by summing 1 divided by k plus its rank in each list where it appears.

The formula is RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the difference between adjacent top ranks and prevents the first position from overwhelming all other evidence. A commonly used starting point is k = 60, but it is a tuning choice rather than a universal truth. Sort documents by the fused score, then fetch the document payloads needed by the application.

  • Use one-based ranks: the first result has rank 1.
  • Assign no contribution from a retriever when a document is absent from its candidate list.
  • Deduplicate by a canonical document or chunk ID before final sorting.
  • Retrieve enough candidates from each source that useful overlap and complementary results can enter the fusion step.

Make fusion observable before making it more complex

Log the dense rank, BM25 rank, and final fused rank for each returned document. These fields make it possible to see whether a result was promoted by agreement between retrievers, rescued by one retriever, or repeatedly absent from both. They are also useful when investigating surprising results involving exact strings or ambiguous natural-language queries.

Evaluate changes with a representative query set and relevance judgments where available. Start with a simple unweighted RRF implementation, then consider source-specific weights only when evaluation shows a consistent need. The operational goal is not to force dense and sparse scores into a common numeric scale; it is to produce a reliable final ordering from complementary retrieval evidence.

  • Record candidate depth and the fusion constant alongside each experiment.
  • Check queries containing product names, IDs, quoted phrases, and paraphrased questions.
  • Watch for duplicate chunks from the same source document dominating the final list.
  • Version retrieval settings so ranking changes can be reproduced and compared.