Why raw dense and sparse scores should not be added

Dense retrieval ranks documents by the relationship between embedding vectors. BM25 ranks documents from term frequency, document length, and corpus-level term statistics. Even when both systems return a number called a score, those values do not share a universal meaning or range.

Adding a dense score directly to a BM25 score can make ranking sensitive to changes that are unrelated to document relevance. A re-embedding, an index rebuild, a BM25 configuration change, or a different query distribution can alter score ranges. The retrieval pipeline may then favor one source simply because its numeric values are larger.

Rank-based fusion is a useful baseline because it asks a narrower question: how highly did each retrieval method rank a document? This preserves evidence from both semantic matches and exact-term matches without assuming that their score scales are compatible.

  • Dense search can retrieve semantically related wording.
  • BM25 can reward exact identifiers, names, and rare query terms.
  • Raw score magnitudes are not inherently comparable across retrieval methods.
  • A fusion strategy should be evaluated on relevance outcomes, not score appearance.

Fuse the top-k lists with Reciprocal Rank Fusion

Run dense retrieval and BM25 retrieval for the same query, then retain a top-k list from each. For every unique document identifier found in either list, calculate an RRF score: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across the lists in which document d appears.

The constant k controls how quickly rank differences decay. A commonly used starting value is 60, but it is a tuning parameter rather than a rule. With a larger value, moving from rank 1 to rank 10 matters less; with a smaller value, the top positions receive more emphasis.

Documents retrieved by both methods gain two contributions, while documents unique to one method can still rank well if they appear near that method's top. This is especially helpful for queries that contain both a conceptual request and an exact token, such as a product name, error code, or internal acronym.

  • Deduplicate candidates by a stable document or chunk identifier.
  • Use one-based ranks: the first result has rank 1.
  • Assign no contribution for a list where the document does not appear.
  • Sort by descending RRF score and return the desired final top-k.

Make fusion observable before making it complex

Begin with an intentionally small, inspectable pipeline: retrieve a fixed number of dense and BM25 candidates, apply RRF, and log each candidate's source ranks. For a fused result, recording dense rank, BM25 rank, and final fusion score makes ranking behavior understandable during debugging.

Evaluate with representative queries rather than only broad natural-language prompts. Include exact-match queries, synonym-heavy queries, short queries, long questions, and queries with ambiguous vocabulary. The goal is to learn where dense search, BM25, and their combination each contribute useful candidates.

If one source consistently introduces irrelevant items, first inspect candidate depth, chunking, metadata filters, and query construction. Weighted or learned fusion can be appropriate later, but a clear unweighted RRF baseline gives those later decisions a reliable point of comparison.

  • Log document ID, dense rank, BM25 rank, and fused rank.
  • Review queries where the top fused result differs from both individual winners.
  • Keep retrieval depth separate from the number of results shown to users.
  • Re-evaluate after changing embeddings, text processing, chunking, or corpus content.