Why raw dense and BM25 scores should not be added blindly

A dense-search score and a BM25 score are produced by different retrieval models and have different distributions. Their numeric ranges can change with the embedding model, index configuration, query wording, document length, and corpus composition. A score of 0.7 from one system does not inherently mean the same thing as a score of 0.7 from another.

Adding those values directly creates an implicit calibration rule: it assumes both scores are comparable and that a fixed weighting remains correct for every query. That assumption often fails on mixed workloads. A natural-language question may benefit from semantic matches, while a query containing a ticket number or function name may depend primarily on lexical matching.

  • Dense retrieval helps with paraphrases, related concepts, and vocabulary mismatch.
  • BM25 helps with exact terminology, proper nouns, codes, and uncommon tokens.
  • Score scales are model- and system-specific, so a global weighting can be fragile.
  • Rank positions are easier to compare across independently retrieved result lists.

Apply reciprocal rank fusion to two candidate lists

Run the same query against dense retrieval and BM25 retrieval, then retain a bounded candidate list from each. For every document d, calculate an RRF score: RRF(d) = Σ 1 / (k + rank_i(d)). The sum includes each result list in which the document appears, and rank_i(d) is the one-based position of the document in list i.

The constant k reduces the effect of small differences near the top of a list. A document ranked highly by both retrievers receives two contributions, while a document found by only one retriever can still be included. This rewards agreement without requiring either system's underlying score to be interpreted as a probability or universal relevance measure.

  • Normalize document identifiers before fusion so the same logical document can be recognized across both lists.
  • Use one-based ranks: the first result has rank 1.
  • Choose a candidate depth that leaves enough material for fusion and any later reranking step.
  • Keep the dense and BM25 raw scores in logs for diagnosis, even if RRF does not use them.

Make fusion observable and query-aware

RRF is deliberately simple, but it should still be evaluated against representative queries and relevance judgments. Segment evaluation by query type: exact identifiers, product terminology, short natural-language questions, long questions, and ambiguous queries. A single aggregate metric can hide the fact that one retriever is carrying an important query segment.

In production, record which source retrieved each fused result, its rank in each source, and its final fused position. These fields make it possible to investigate surprising results. They also support later refinements, such as changing candidate depths, using source-specific weights, or routing clearly identifier-heavy queries through a lexical-first path.

  • Measure recall at the candidate stage before judging final ranked quality.
  • Inspect overlap between dense and BM25 lists; low overlap is not automatically a problem.
  • Test queries with identifiers and rare terms separately from semantic questions.
  • Version the embedding model, corpus snapshot, and fusion settings alongside evaluation results.