Why raw dense and BM25 scores should not be added

A dense retrieval score expresses similarity in an embedding space. A BM25 score is derived from term statistics, document length, and query-term frequency. Even when both are useful relevance signals, their numeric ranges and distributions are not inherently comparable.

Adding those scores directly creates a hidden tuning problem. A multiplier that works for one corpus, embedding model, or query type may behave differently after documents change or a new model is introduced. Rank-based fusion avoids requiring a shared score meaning.

  • Use dense retrieval for semantic matches, paraphrases, and conceptually related language.
  • Use BM25 for exact terms, identifiers, uncommon names, and literal phrasing.
  • Treat each retriever's score as local to that retriever unless it has been explicitly calibrated.

Fuse two ranked lists with RRF

Run the same user query through the dense and sparse paths, requesting a sufficiently deep candidate list from each. For every document that appears in either list, assign an RRF contribution based on its one-based rank: 1 divided by k plus the rank. Sum the contributions across lists.

The constant k reduces the advantage of a first-place result over other high-ranked results. A commonly used starting value is 60, but it is a policy choice rather than a universal optimum. The important property is that a document supported by both retrieval methods rises without requiring their raw scores to align.

  • Formula: RRF(document) = Σ 1 / (k + rank_i(document)).
  • Include a contribution only when the document appears in retrieval list i.
  • Deduplicate by a stable document ID before returning the final ranking.
  • Keep the original dense and BM25 ranks in diagnostics for later analysis.

Make the fusion step observable and testable

Start with a small set of representative queries: exact product names, error messages, broad conceptual questions, abbreviations, and queries that mix natural language with identifiers. Review the dense list, the BM25 list, and the fused list separately. This reveals whether an issue originates in indexing, query construction, candidate depth, or fusion.

Candidate depth matters because RRF can only promote documents that were retrieved by at least one source. If relevant documents regularly appear just beyond the cutoff in one list, increase that list's candidate count before changing k. Evaluate changes against a fixed relevance set so tuning remains tied to observed retrieval quality.

  • Log query text, source ranks, fused rank, and document ID for sampled searches.
  • Record whether a returned result came from dense retrieval, BM25, or both.
  • Test candidate-depth changes independently from k changes.
  • Re-run the evaluation set after embedding, analyzer, or corpus updates.