Why raw dense and sparse scores should not be added

A dense retrieval score is produced by a vector similarity function and depends on embedding construction, normalization, and index behavior. A BM25 score is based on term frequency, document frequency, and document-length normalization. Even when both systems return a numeric score, the numbers are not automatically on a common scale.

Adding those values directly can make one retriever dominate for accidental reasons. A query may produce a narrow range of vector scores and a wide range of BM25 scores, or the reverse. Score distributions can also change as documents, embeddings, analyzers, or index settings change.

  • Treat each engine's score as meaningful primarily within that engine's own result list.
  • Avoid choosing a dense-versus-sparse weight solely because one score range appears larger.
  • Inspect rankings for representative queries before introducing score-based blending.
  • Use a fusion method based on rank when score calibration is unavailable.

Fuse two top-k lists with Reciprocal Rank Fusion

RRF assigns a contribution to each document based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank for every list in which it appears: RRF(d) = Σ 1 / (k + rank_i(d)). A document returned by both dense and BM25 retrieval is naturally promoted, while a document ranked highly by only one method can still remain competitive.

The constant k reduces the difference between nearby ranks. A commonly used starting value is 60, but it is a tuning parameter rather than a universal rule. The important operational property is that RRF needs ordered result lists, not comparable raw scores.

  • Request a candidate list from S3 Vectors using the query embedding.
  • Request a candidate list from Quickwit BM25 using the query text.
  • Deduplicate candidates using a stable document or chunk identifier.
  • Sum each candidate's rank-based RRF contributions, then sort by the fused score.

Choose candidate depth and evaluate the failure cases

Fusion can only promote documents that appear in at least one input list. If an answer-bearing chunk is absent from both candidate sets, RRF cannot recover it. Candidate depth therefore matters: retrieve enough documents from each path to create useful overlap and complementary coverage before returning a smaller final set or sending candidates to a reranker.

Evaluate hybrid retrieval with a query set that reflects actual traffic. Include queries with exact identifiers, spelling variations, short natural-language questions, domain-specific terminology, and queries where users describe a concept without using the document's exact wording. Review not only aggregate relevance but also which retrieval path supplied each successful result.

  • Log the dense rank, BM25 rank, and fused rank for returned candidates.
  • Measure whether relevant documents appear in either candidate list before evaluating fusion.
  • Tune candidate depth and the RRF constant against judged queries, not anecdotal examples alone.
  • Keep the dense and sparse paths independently observable so regressions are easier to isolate.