Why raw dense and BM25 scores should not be directly compared

A dense-search score and a BM25 score are produced by different ranking models. Their scales, distributions, and sensitivity to query length are not inherently comparable. A score of 0.7 from one system does not have a universal relationship to a score of 12 from another.

Adding raw scores together can therefore create unstable behavior. A small change in embedding model, index configuration, document length, or BM25 analyzer may shift one score distribution enough to dominate the combined ranking, even when the underlying relevance has not meaningfully changed.

  • Use separate dense and sparse retrieval requests for the same query.
  • Request a bounded candidate set from each retriever, such as the top N results.
  • Keep document identifiers consistent across both result lists so duplicates can be merged.
  • Treat each retriever's ranking as useful evidence, not as a universally calibrated score.

Merge candidate lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in every ranking where it appears. For a document d, the fused score is the sum of 1 divided by k plus its rank: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the difference between neighboring ranks and keeps the fusion focused on consistent high placement.

A document returned by both dense search and BM25 receives two contributions. A document that is exceptionally strong in only one list can still appear, while a document that ranks well across both signals tends to rise. This is valuable because it does not require a shared score scale.

  • Choose a fixed k and keep it explicit in configuration; 60 is a commonly used starting point, not a universal optimum.
  • Use one-based ranks: the first result has rank 1.
  • Deduplicate by stable document ID before presenting results.
  • Optionally assign a weight to each list when a product has a deliberate preference for dense or lexical recall.

Make fusion observable before tuning it

RRF is simple enough to implement, but it should still be evaluated with representative queries. Include exact-token queries such as product names or error codes, semantic paraphrases, short ambiguous questions, and longer natural-language requests. The goal is to learn whether the fused list retrieves the right candidates for the workload, not merely whether it looks balanced.

Log the dense rank, BM25 rank, and fused rank for returned documents. Those fields make failures easier to diagnose. If relevant documents appear only in one candidate list, increasing that list's retrieval depth may help; if irrelevant exact matches crowd out useful passages, review the sparse-query analysis and document chunking strategy before changing fusion weights.

  • Create a small labeled query set from real user intents and known relevant documents.
  • Measure candidate recall before evaluating final ranking quality.
  • Inspect overlap between dense and BM25 candidate lists; very low overlap is informative, not automatically a problem.
  • Version retrieval settings, including candidate depth, k, weights, analyzers, and embedding configuration.