Why raw-score merging is usually fragile

A dense retrieval score is derived from the relationship between an embedded query and embedded documents. A BM25 score is driven by term frequency, document frequency, and length normalization. Even when both scores increase with relevance, their numeric ranges, distributions, and sensitivity to query wording are different.

Adding or averaging those values assumes they are calibrated to a common meaning. In practice, a query containing a rare product code may produce a sharply peaked BM25 ranking, while a broad conceptual query may yield denser semantic scores. A fixed weight that appears sensible for one query class can distort another.

  • Do not assume a score of 0.8 from dense retrieval has the same interpretation as a BM25 score of 0.8.
  • Per-query score normalization can help, but it introduces choices about distributions, outliers, and missing candidates.
  • Start with a fusion method that needs ordering, not comparable raw scores.

Fuse independent result lists with RRF

Run dense search against your vector collection and BM25 search against your sparse index, requesting a sufficiently deep candidate list from each. For every document returned by either path, assign an RRF contribution based on its position in each list. Documents appearing in both lists accumulate credit; documents highly ranked by one method can still remain competitive.

The common formula is RRF(d) = sum over result lists of 1 / (k + rank(d)). Here, rank starts at 1 and k is a positive constant that reduces the influence of small position differences near the top of a list. The output is sorted by its summed RRF score.

  • Deduplicate by a stable document identifier before calculating the final order.
  • Use the same document scope and filters for dense and BM25 retrieval whenever possible.
  • Choose candidate depths larger than the final page size so fusion has meaningful overlap and alternatives.
  • Treat k as a tuning parameter; document the chosen value and evaluate it on representative queries.

Make fusion observable before making it clever

Log the rank and source membership of each fused result: dense-only, BM25-only, or present in both. This makes hybrid behavior inspectable. If exact identifiers disappear, inspect the BM25 candidate depth and filtering. If conceptually related content is absent, inspect chunking, embeddings, and dense candidate depth before changing fusion weights.

Evaluate with a query set that reflects actual retrieval tasks, including identifier lookups, paraphrased questions, acronym-heavy queries, and queries with ambiguous terms. Compare dense-only, BM25-only, and RRF lists using human judgments or task-specific success criteria. The goal is not to make every query hybrid by default; it is to understand which signal contributes useful evidence.

  • Record the original rank from each retrieval path alongside the fused rank.
  • Review failures by query type rather than relying only on one aggregate metric.
  • Keep a dense-only and sparse-only fallback path for diagnosis.
  • Change one variable at a time: candidate depth, k, chunking, or filtering.