Why raw dense and sparse scores should not be added directly

A dense retrieval score is produced from an embedding similarity calculation, while a BM25 score is derived from term frequency, document frequency, and document length effects. Even when both scores increase with relevance, their numerical ranges and distributions are not inherently compatible.

Adding those values directly creates a hidden calibration problem. A change in embedding model, index settings, document chunking, or corpus composition can alter one score distribution and quietly change the balance of the combined ranking. Rank-based fusion avoids depending on that numerical alignment.

In a Talqora-oriented architecture, dense candidates can come from regional S3 Vectors and sparse candidates from Quickwit BM25. Treat each result set as an ordered signal: the position of a document in each list is the input to fusion, not the raw score returned by either retrieval method.

  • Dense search helps with paraphrases, related concepts, and natural-language questions.
  • BM25 helps with exact product names, error codes, acronyms, and uncommon terms.
  • Independent score scales are normal; they are not evidence that one retriever is wrong.
  • Use a stable document or chunk identifier to recognize the same item across both result lists.

Apply Reciprocal Rank Fusion to the two candidate lists

For each candidate document, calculate an RRF score by summing a contribution from every list where it appears. The usual form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in result list i and k is a positive constant.

The constant k dampens the advantage of being at the very top of a single list. A document that ranks well in both dense and BM25 results can therefore outrank a document that appears only once at rank one, depending on the ranks and chosen k. Keep k explicit in configuration so relevance behavior can be reviewed and changed deliberately.

Fetch a bounded candidate set from each retriever, deduplicate by the canonical identifier, compute the fused score, and sort descending. Retain source ranks alongside the final result; they are valuable when investigating why a passage was retrieved.

  • Request the same candidate depth from both retrieval paths as an initial operational baseline.
  • Assign ranks after each source list has been sorted by its own native relevance score.
  • Give an absent document no contribution from that retrieval path.
  • Break equal fused scores deterministically, such as by best individual rank and then a stable identifier.

Evaluate fusion with query classes, not one aggregate number

RRF is simple, but it is still a retrieval policy. Evaluate it against representative query classes: exact identifier lookups, short keyword queries, conceptual questions, multi-part questions, and queries that mix a name with a natural-language description. A single aggregate relevance result can conceal regressions in an important class.

Inspect disagreements between the dense and BM25 lists. If BM25 finds a precise document that dense retrieval misses, the issue may be embedding coverage or chunk content. If dense retrieval finds useful paraphrases absent from BM25, that confirms the value of retaining semantic retrieval. Fusion is most useful when the two paths have complementary failures.

Start with RRF before introducing learned score calibration or complex weighting. A transparent baseline makes later adjustments easier to justify: changes can be compared against a known policy, and debugging can begin with two source ranks rather than an opaque combined score.

  • Record the dense rank, BM25 rank, fused rank, and canonical result identifier for evaluation queries.
  • Review top results manually for high-value query classes before changing fusion settings.
  • Test chunk-level retrieval separately from document-level presentation.
  • Keep retrieval fusion distinct from any later reranking stage so each decision can be evaluated independently.