Why raw dense and BM25 scores should not be added directly

A dense retriever returns similarity-oriented scores derived from vector distance or similarity. Quickwit BM25 returns lexical relevance scores based on term frequency, document frequency, and field statistics. Even when both are useful signals, a score of 0.8 from one retrieval path does not inherently mean the same thing as a score of 0.8 from the other.

Adding uncalibrated scores can make ranking sensitive to index settings, query composition, score distributions, and changes in either retrieval system. A safer first design is to preserve each system’s ordering and combine ranks rather than raw numeric values.

  • Use dense retrieval for semantic intent and paraphrases.
  • Use BM25 for exact phrases, product names, codes, and uncommon terms.
  • Treat score scales as retrieval-system-specific unless they have been explicitly calibrated.

Apply reciprocal rank fusion to two candidate lists

Run the same user query through the dense path backed by regional S3 Vectors and the sparse path backed by Quickwit BM25. Each path should return a bounded candidate list using a shared, stable document identifier. The application can then merge those lists with RRF.

For each document, RRF adds a contribution based on its position in every list where it appears: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the gap between adjacent top positions and prevents the first few ranks from overwhelming all other evidence. A commonly used starting point is k = 60, but it should be evaluated against the relevance behavior of the corpus.

  • Retrieve a fixed depth from both systems before fusion, such as the top N candidates from each path.
  • Assign rank 1 to the first result in each individual list.
  • Sum RRF contributions for documents returned by one or both paths.
  • Sort by the fused score and retain the final retrieval set for downstream use.

Make fusion reliable with document identity and evaluation

RRF only works cleanly when dense and sparse indexes refer to the same logical documents. Store a canonical document ID in both indexing pipelines, and decide whether the retrieval unit is a full document, section, chunk, or another stable unit. If a BM25 hit identifies a document while a vector hit identifies a chunk, fusion can produce misleading duplicates unless that relationship is resolved before ranking.

Evaluate hybrid retrieval with representative queries rather than assuming that combining two methods is always better. Include exact-lookup queries, natural-language questions, acronym-heavy searches, and queries with ambiguous terms. Review not only aggregate metrics but also the failure cases where one retrieval path contributes results the other misses.

  • Deduplicate by canonical ID before presenting results.
  • Keep index filters and access-control constraints consistent across both retrieval paths.
  • Log per-query dense rank, BM25 rank, and fused rank for debugging.
  • Re-evaluate candidate depth and k when corpus content or indexing strategy changes.