Why raw dense and sparse scores should not be added

A dense retrieval score reflects the similarity function and embedding representation used by the vector index. A BM25 score reflects term frequency, document frequency, document length, and the query terms. Even when both scores increase with relevance, their numeric ranges and distributions need not mean the same thing.

Adding raw scores can make one retriever dominate simply because it emits larger numbers. That behavior may change when documents are added, analyzers are adjusted, embeddings are replaced, or search settings evolve. Ranking-based fusion avoids depending on a stable cross-system score scale.

  • Treat each retriever's score as meaningful within that retriever, not automatically across retrievers.
  • Inspect ranked results separately before deciding how to combine them.
  • Avoid choosing score weights solely because one score range appears numerically larger.

Fuse two ranked lists with RRF

RRF assigns every document a contribution based on its position in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) for every list in which d appears. The rank is one-based, and k is a positive constant that reduces the difference between nearby positions.

In a Talqora-oriented retrieval flow, obtain a dense candidate list from regional S3 Vectors and a sparse candidate list from Quickwit BM25. Deduplicate document IDs, calculate an RRF score for each candidate, sort by that fused score, and return the leading documents. A document found by both retrievers gains two contributions, while a document ranked highly by either system can still remain competitive.

  • Request a candidate depth larger than the number of results you plan to show.
  • Use a consistent document identifier across dense and sparse indexes.
  • Start with one shared k value and evaluate changes using representative queries.
  • Define a deterministic tie-breaker, such as a document ID or recency field, when fused scores match.

Evaluate fusion using query failure modes

Do not evaluate hybrid retrieval only on broad natural-language questions. Include queries that expose complementary behavior: product names, error codes, quoted phrases, abbreviations, paraphrases, and questions whose answer uses wording different from the query. These cases reveal whether fusion is preserving the strengths of both retrieval methods.

Log the dense rank, BM25 rank, fused rank, and selected document IDs for a small reviewed query set. When an expected document falls, determine whether it was absent from both candidate lists, present too low in one list, or displaced during fusion. Each outcome points to a different fix: candidate depth, indexing quality, query construction, or the fusion policy.

  • Track whether relevant documents are retrieved by dense search, BM25, or both.
  • Review queries where exact terminology matters separately from semantic paraphrases.
  • Keep an evaluation set stable when comparing changes to embeddings, analyzers, or fusion parameters.
  • Recheck filters and access constraints before fusion so ineligible documents cannot enter the final ranking.