Why raw-score blending is fragile

A dense retrieval score and a BM25 score may both indicate relevance, but they are produced by different ranking methods. Their ranges, distributions, and sensitivity to query length can differ. Adding them together with a fixed weight can make ranking behavior difficult to predict.

This becomes especially noticeable across mixed query types. A natural-language question may benefit from dense retrieval, while a query containing a product code, error message, or quoted phrase may be best served by BM25. A score threshold that seems reasonable for one path may be unsuitable for the other.

  • Do not assume equal numeric scores represent equal relevance.
  • Avoid choosing dense-versus-sparse weights solely from a few example queries.
  • Treat the two result lists as ranked evidence, not as directly interchangeable score values.

Apply reciprocal rank fusion

Run the same user query through the dense and sparse retrieval paths. Request a sufficiently deep candidate list from each path, then deduplicate documents by a stable document identifier. For each document, add a contribution based on its rank in each list.

The common RRF formula is: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the advantage of a single top-ranked placement and makes the fusion less sensitive to small rank changes. A document appearing in both lists receives contributions from both, even if it is not first in either one.

  • Use one-based ranks: the first result has rank 1.
  • Choose a fixed k and keep it stable while evaluating behavior; 60 is a commonly used starting convention, not a universal optimum.
  • Fuse only the top N results from each retriever to bound application work.
  • Use a stable ID for deduplication before calculating the final order.

Preserve observability and tune with real queries

Return or log the fused rank alongside each document's dense rank and BM25 rank. This makes it possible to explain why a result appeared and to identify cases where one retrieval path dominates. It also helps distinguish ranking issues from indexing, chunking, or metadata-filtering issues.

Evaluate with a small, maintained set of representative queries before changing candidate depth or fusion settings. Include semantic questions, exact-name lookups, identifier-heavy searches, and short ambiguous queries. The goal is not to make every query look the same, but to ensure each retrieval signal can contribute when it is useful.

  • Log document ID, final fused rank, and rank from each retrieval path.
  • Inspect queries where relevant documents appear in only one candidate list.
  • Keep candidate depth, filters, and fusion settings versioned with application changes.
  • Measure relevance with human judgments or task outcomes rather than score values alone.