Why raw dense and sparse scores should not be added

Dense retrieval ranks documents by the relationship between an embedded query and embedded content. BM25 ranks documents from term occurrence, term rarity, and document-length normalization. A score of 0.8 from one system does not inherently mean the same thing as a score of 0.8 from the other.

Adding uncalibrated scores can make a hybrid system fragile. A change in embedding model, chunking policy, BM25 analyzer, or corpus composition may change score distributions even when relevance remains stable. Rank-based fusion avoids treating those distributions as directly comparable.

  • Use dense retrieval for semantic paraphrases and concept-level matches.
  • Use BM25 for exact identifiers, uncommon names, quoted phrases, and lexical constraints.
  • Treat each retrieval path as a ranked candidate generator rather than a shared scoring scale.

Fuse the two result lists with RRF

RRF assigns each document a contribution based on its position in each ranked list. For a document d, compute RRF(d) = sum of 1 / (k + rank_i(d)) across the lists where d appears. The rank starts at 1, and k is a positive constant that reduces the advantage of being first by a small margin.

For example, request a candidate list from dense search and another from BM25, deduplicate document IDs, calculate the RRF score for each ID, and sort descending. A document that appears near the top of both lists will usually rise above a document that appears only in one list.

  • Start with the same candidate depth for dense and sparse retrieval, such as the top 50 from each path.
  • Use a fixed k consistently while evaluating relevance; k = 60 is a commonly used starting point, not a universal optimum.
  • Assign no contribution when a document is absent from a list.
  • Keep source ranks and fusion scores in logs for debugging and evaluation.

Make fusion operationally useful

RRF is most useful when the documents returned by both paths have a stable identity. Fuse at the document or chunk ID used by the application, then fetch the stored content and metadata only for the final ranked set. If a source document has many chunks, decide whether ranking should happen at chunk level or after collapsing chunks to a parent document.

Evaluate the fused list against representative queries, especially queries with product codes, abbreviations, spelling variation, and natural-language descriptions. Inspect failures by source: a missed exact match points toward sparse retrieval or analysis, while a missed paraphrase points toward embedding, chunking, or dense candidate depth.

  • Apply metadata filters consistently to both retrieval paths before fusion.
  • Deduplicate repeated chunks before presenting results to users.
  • Record whether each final result came from dense retrieval, BM25, or both.
  • Re-evaluate after changing embeddings, analyzers, chunk sizes, or corpus structure.