Why raw-score blending is fragile

Dense retrieval and BM25 answer different matching questions. Dense search ranks vectors by semantic proximity, while BM25 rewards term occurrences using corpus-aware weighting. Their score ranges, distributions, and sensitivity to query length are not inherently aligned.

Adding or averaging raw scores therefore requires calibration choices that can change as embeddings, document collections, analyzers, or query traffic change. A rank-based method avoids assuming that a score of one system has the same meaning as a score from the other.

  • Use dense retrieval for paraphrases, related concepts, and vocabulary mismatch.
  • Use BM25 for exact identifiers, product names, error messages, and rare terms.
  • Treat the two result lists as independent evidence rather than comparable score scales.

Apply reciprocal rank fusion to both candidate lists

Run the same user query through the dense path backed by regional S3 Vectors and the sparse path backed by Quickwit BM25. Keep a bounded candidate list from each path, then assign every returned document an RRF contribution based on its position in that list.

For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)), summing over the result lists where d appears. The constant k reduces the gap between adjacent top ranks; it should be chosen deliberately and kept visible in retrieval configuration.

  • Use stable document IDs so dense and sparse results can be joined correctly.
  • Define whether ranks start at 1 and apply that convention consistently.
  • Deduplicate documents before returning the final fused ranking.
  • Retain per-source ranks in logs to make surprising results explainable.

Evaluate retrieval failures by query class

RRF is simple, but it is still a retrieval policy. Evaluate it with representative queries rather than relying only on a blended aggregate metric. Exact lookup queries, natural-language questions, multi-concept queries, and short ambiguous queries often expose different failure modes.

Inspect cases where only one retrieval path finds the desired document. Those examples help determine candidate-list depth, the fusion constant, and whether query routing or filtering is needed before fusion. The goal is not to force equal participation; it is to preserve useful evidence from both retrieval modes.

  • Create a small judged set with expected documents or relevance labels.
  • Compare dense-only, BM25-only, and fused rankings for the same queries.
  • Review zero-result and low-confidence query patterns separately.
  • Version retrieval settings alongside embedding and indexing changes.