Why dense and sparse scores should not be added blindly

A dense retriever typically returns a similarity score derived from embedding vectors, while BM25 returns a lexical relevance score based on term frequency and document statistics. Even when both lists are useful, their numeric scales do not inherently mean the same thing. A score of 0.8 from one system is not automatically equivalent to a score of 0.8, or 8.0, from another.

Raw-score addition therefore introduces a calibration problem. Score distributions can shift when embeddings change, document collections grow, analyzers are adjusted, or query language changes. Rank-based fusion instead asks a simpler question: how highly did each retriever place a document?

  • Dense retrieval can connect paraphrases and related concepts.
  • BM25 can preserve exact matches for IDs, acronyms, codes, and rare terms.
  • Each retriever produces an ordered candidate list.
  • Fusion can operate on ranks without requiring shared score semantics.

Fuse candidate lists with Reciprocal Rank Fusion

For every document d, RRF sums a small contribution from each result list in which d appears: RRF(d) = Σ 1 / (k + rank_i(d)). The rank starts at 1 for the top result, and k is a positive constant that reduces the difference between adjacent positions. Documents surfaced near the top by both dense and sparse retrieval accumulate more evidence.

A useful implementation pattern is to request a bounded candidate set from S3 Vectors and a bounded candidate set from Quickwit BM25, deduplicate by a stable document or chunk identifier, compute the fused score, then sort descending. Keep source ranks alongside the fused result during development; they make ranking behavior inspectable.

  • Use the same chunk or document ID in both indexes.
  • Choose a fixed candidate depth for each retriever before fusion.
  • Treat a missing document as contributing zero from that retriever.
  • Start with one shared k value and change it only with an evaluation set.

Evaluate the failures that hybrid retrieval is meant to fix

Do not evaluate hybrid retrieval only on broad natural-language questions. Include queries with known lexical anchors: ticket numbers, API field names, version strings, policy clauses, error text, and organization-specific vocabulary. These are cases where sparse retrieval may rescue a result that semantic similarity ranks too low.

Also include paraphrases and underspecified questions, where dense retrieval may find relevant content despite vocabulary mismatch. Review not only whether a relevant result appears, but where it appears in the final list. The point of fusion is usually to improve the quality of the first few candidates passed to a downstream reader or reranker.

  • Create a small query set from real retrieval tasks and known edge cases.
  • Record relevance judgments at the document or chunk level.
  • Compare dense-only, BM25-only, and fused rankings side by side.
  • Inspect queries where fusion hurts; duplicated chunks and inconsistent IDs are common causes.