Why raw-score blending is fragile

Dense retrieval and BM25 answer different signals. Dense search ranks documents by embedding similarity, which can surface semantically related material even when it does not repeat the query’s wording. BM25 ranks term matches using lexical statistics, making it valuable for identifiers, product names, error messages, and uncommon phrases.

Their scores should not be assumed to share a common meaning. A dense similarity score and a BM25 score arise from different calculations and can change distribution as the corpus, query wording, or search configuration changes. Adding them directly therefore introduces a tuning problem: a weight that appears reasonable for one query set may over-favor one retriever on another.

  • Use dense retrieval to capture semantic intent.
  • Use BM25 to preserve exact lexical evidence.
  • Treat each retriever’s score scale as retriever-specific.
  • Prefer a fusion method that relies on ordering when calibration is unavailable.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), summing across retrieval lists where the document appears. The constant k reduces the advantage of a single first-place result and makes the contribution decline gradually with rank.

In practice, request a candidate list from S3 Vectors and a candidate list from Quickwit BM25, normalize document identifiers, and combine duplicate documents by summing their rank contributions. Sort the merged candidates by the resulting RRF score. A document found by both retrieval paths is rewarded, while a strong result from only one path can still remain competitive.

  • Choose the same candidate depth initially, such as the top N results from each retriever.
  • Use a stable canonical document ID before deduplicating results.
  • Start with a fixed k and evaluate it against representative queries.
  • Keep the original dense rank and BM25 rank for debugging.

Evaluate fusion by query type, not only by averages

Build a small evaluation set that reflects how people actually search. Include conceptual questions, exact identifiers, quoted phrases, abbreviated terms, and queries that mix an entity name with a natural-language request. For each query, define one or more relevant documents and compare dense-only, BM25-only, and fused rankings.

Review failures at the query level. If an exact error code falls after loosely related semantic matches, inspect whether the BM25 candidate depth is too shallow or whether the fusion setting is too conservative. If a conceptually relevant document disappears because it lacks the query’s vocabulary, inspect dense candidate depth and embedding quality. Fusion is a retrieval policy that should be revised with evidence, not a substitute for evaluation.

  • Track whether relevant documents appear in the candidate set before judging final rank.
  • Segment results into lexical, semantic, and mixed-intent queries.
  • Log per-retriever ranks alongside the fused rank.
  • Re-test after corpus changes, document chunking changes, or query-pattern shifts.