Why dense and sparse scores should not be added blindly

A dense retriever and a BM25 index produce scores for different reasons. A dense score reflects a vector similarity calculation, while BM25 is based on term frequency, document frequency, and document-length normalization. Even when both scores increase for better matches within their own result lists, their scales and distributions need not align.

Adding raw scores can therefore create accidental weighting. A system with numerically larger scores may dominate the merged ranking regardless of whether it is producing the more useful candidates for a given query. Score normalization can be appropriate when it is measured and maintained carefully, but it introduces assumptions about score distributions.

Rank-based fusion avoids that immediate comparability problem. It asks a narrower question: how highly did each retriever rank a document among its own candidates?

  • Use dense retrieval for semantic similarity and paraphrased intent.
  • Use BM25 for exact tokens, rare terms, names, and identifiers.
  • Treat raw scores as retriever-specific unless they have been deliberately calibrated.
  • Keep each retriever’s candidate list available before applying a merge rule.

Apply Reciprocal Rank Fusion at the application layer

RRF assigns a contribution to a document from each ranked list in which it appears. For a document d, the fused score is the sum of 1 divided by k plus the document’s rank in each list: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the gap between adjacent positions and prevents the very first rank from overwhelming the rest of the list.

For each query, request a candidate set from the dense path and a candidate set from the sparse path. Deduplicate candidates using a stable document identifier, calculate the RRF total for every identifier, then sort descending by that total. A document returned by both retrievers gains two contributions; a document highly ranked by only one can still remain competitive.

RRF does not require the dense and sparse paths to expose matching score semantics. It only requires ordered results and a consistent definition of rank, such as rank 1 for the first result.

  • Retrieve the same top-N depth from both paths as an initial baseline.
  • Use a stable canonical ID to deduplicate documents across result lists.
  • Assign ranks after any per-retriever filtering that affects visible candidates.
  • Record the source ranks alongside the fused result for debugging.

Make fusion observable before making it complicated

A merged ranking is easier to improve when its inputs are inspectable. For sampled queries, log the dense rank, BM25 rank, fused rank, and whether a result appeared in one or both lists. This makes it possible to distinguish useful agreement from cases where one retriever contributes irrelevant candidates.

Start with a fixed k and a fixed candidate depth, then evaluate representative query classes separately. Queries containing quoted text, part numbers, product names, or error messages may lean on sparse retrieval. Broad natural-language questions may benefit more from dense retrieval. The goal is not to declare one approach universally better, but to understand where each contributes.

If later evaluation justifies query-dependent behavior, keep the rule explicit. For example, an application may detect identifier-like queries and adjust candidate allocation before fusion. Such changes should be tested against held-out queries, because heuristics can improve one query type while reducing coverage elsewhere.

  • Create a small relevance set from real query patterns before tuning constants.
  • Inspect failures by query class, not only by an aggregate metric.
  • Version fusion rules so ranking changes can be reproduced.
  • Preserve per-result retrieval metadata for audits and troubleshooting.