Why raw-score blending is fragile

A dense search score and a BM25 score usually have different meanings and ranges. Dense similarity depends on the embedding model, vector representation, and similarity measure. BM25 depends on term frequency, document length, corpus statistics, and query terms. Adding these scores together without careful calibration can cause one retriever to dominate for reasons unrelated to result quality.

Score distributions also change over time. Re-embedding a corpus, changing chunk sizes, adjusting BM25 analysis, or indexing a new content type can shift score ranges. A blending weight that appeared reasonable during an initial test may become poorly calibrated after an operational change.

  • Do not assume a dense score of 0.8 has a universal relationship to a BM25 score of 8.
  • Avoid tuning a single weighted-sum formula against a small set of hand-picked queries.
  • Treat ranking order as more portable than raw score magnitude when combining independent retrievers.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its rank in each result list. For a document d, compute RRF(d) as the sum of 1 divided by k plus its rank, across the dense and sparse lists. The constant k reduces the difference between nearby positions and prevents the first-ranked item from overwhelming all other candidates.

In a Talqora retrieval flow, an application can query regional S3 Vectors for a dense candidate list and Quickwit BM25 for a sparse candidate list. Deduplicate candidates by a stable document or chunk identifier, calculate the fused score, and return the highest-ranked merged results. This keeps each backend responsible for the retrieval method it is designed to perform.

  • Retrieve a bounded candidate set from both dense and BM25 search.
  • Use a stable ID shared by vector records and sparse-indexed records for deduplication.
  • Choose a fixed k initially, then evaluate it with representative queries before changing it.
  • Preserve per-source rank and score in logs for debugging, even if the client sees only the fused rank.

Evaluate disagreements, not just top results

The most useful hybrid-search test cases are often the queries where dense and sparse retrieval disagree. Exact product codes and quoted error text can reveal whether BM25 is contributing enough. Broad natural-language questions, paraphrases, and vocabulary mismatches can reveal whether dense retrieval is finding useful semantic candidates.

Review results at the chunk level as well as the document level. If multiple chunks from one source occupy the fused top results, consider a post-fusion diversification rule or a document-level cap. Fusion improves candidate selection, but it does not replace decisions about chunking, metadata filters, authorization, or final presentation.

  • Build a small evaluation set containing exact-match, paraphrase, acronym, and mixed-intent queries.
  • Record whether a relevant result came from dense retrieval, BM25, or both.
  • Inspect queries with no overlap between candidate lists; they often expose indexing or query-analysis gaps.
  • Apply metadata and access-control filters consistently to both retrieval paths before fusion.