Why raw-score blending is fragile

A dense retriever and a BM25 retriever produce scores with different meanings. A vector similarity score depends on embedding behavior and similarity configuration, while BM25 reflects term frequency, document frequency, and length normalization. Adding those values together without calibration can make one retrieval path dominate for reasons unrelated to relevance.

Score distributions can also shift as documents, vocabulary, chunking rules, or embedding models change. A weighting scheme that looks reasonable on one evaluation set may become unreliable after an index refresh. Rank-based fusion avoids requiring a universal interpretation of either score.

  • Use dense retrieval for semantic matches and paraphrased questions.
  • Use BM25 for literal terms, IDs, quoted phrases, and uncommon vocabulary.
  • Treat each system's returned ordering as useful evidence even when its numeric scores are not comparable.

Fuse result lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each result list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is its one-based rank in retrieval list i. Sum the contribution across the dense and BM25 lists, then sort documents by the resulting total.

The constant k reduces the impact of the very first positions and makes the fusion less sensitive to small rank changes. A commonly used starting point is k = 60, but it should be treated as a tunable parameter rather than a universal default. The important property is that a document appearing near the top of either list receives meaningful credit.

  • Retrieve a candidate list from S3 Vectors for the query embedding.
  • Retrieve a candidate list from Quickwit BM25 using the original text query.
  • Deduplicate results by a stable document or chunk ID.
  • Sum RRF contributions and return the highest-ranked merged candidates.

Make fusion observable before making it complex

Log which retrieval path contributed each final result, along with its dense rank, sparse rank, and fused rank. This makes hybrid behavior explainable: an exact error code may arrive through BM25 alone, while a relevant troubleshooting passage may rise because it appears in both lists.

Evaluate with a query set that reflects actual usage rather than only broad natural-language questions. Include acronym-heavy queries, identifiers, misspellings, paraphrases, and multi-concept requests. If RRF underperforms for a known query class, investigate candidate depth, chunk boundaries, metadata filtering, and query construction before introducing score normalization or a learned reranker.

  • Keep the source-specific ranks in retrieval logs.
  • Measure whether relevant items appear in either candidate list before judging fusion.
  • Test dense-only, BM25-only, and fused retrieval against the same labeled queries.
  • Apply metadata constraints consistently to both retrieval paths when the application requires them.