Why raw dense and BM25 scores should not be mixed

A dense-search score and a BM25 score are not naturally interchangeable. Their values arise from different models, normalization choices, corpus statistics, and query behavior. A rule such as “add the scores” can look reasonable in testing yet become unstable when document lengths, query types, or embedding models change.

Rank positions are easier to compare because both retrieval paths produce an ordered list. RRF uses those positions to reward documents that appear near the top of either list, with an additional boost for documents retrieved well by both. This lets an application combine results from Talqora’s regional S3 Vectors dense-search path and Quickwit BM25 sparse-search path without assuming their scores share a common scale.

  • Use dense retrieval for semantic similarity and paraphrased language.
  • Use BM25 for literal terms, product codes, names, and rare vocabulary.
  • Keep each retriever’s native score for debugging, but use rank for fusion.
  • Treat fusion as an application-level retrieval policy that can be tested and revised.

Fuse two candidate lists with Reciprocal Rank Fusion

For each query, request a candidate list from dense search and another from BM25. Deduplicate documents by a stable document identifier. Then assign each document a fused score by summing 1 divided by k plus its rank in every list where it appears. The constant k reduces the difference between adjacent top ranks and prevents a single rank-one result from overwhelming all other evidence.

For example, if a document is ranked 2 by dense retrieval and 5 by BM25, its score is 1/(k+2) + 1/(k+5). A document returned only at rank 1 by BM25 receives 1/(k+1). The exact value of k is a policy choice; select it through evaluation rather than assuming one value is universally best.

  • Retrieve a fixed candidate depth from each source before fusion.
  • Use the same canonical ID when deduplicating dense and sparse hits.
  • Sort documents by descending fused score after summing rank contributions.
  • Preserve source ranks and native scores in logs to explain unexpected results.

Evaluate the policy on the queries users actually ask

Hybrid retrieval should be evaluated as a ranking policy, not just as an infrastructure configuration. Build a small query set from search logs, support questions, documentation lookups, or domain-specific tasks. Label useful documents where possible, then compare dense-only, BM25-only, and RRF outputs at the result depth your interface presents.

Pay particular attention to query classes. An exact error code may favor BM25, while a natural-language question may benefit from dense retrieval. When RRF performs poorly, inspect whether one source returned too few candidates, whether documents lack useful text fields, or whether the query needs routing instead of universal fusion. A reliable system often combines broad hybrid coverage with targeted exceptions for known query patterns.

  • Measure relevance at practical cutoffs such as the first few displayed results.
  • Slice evaluation by query type: exact identifiers, short keywords, and natural-language questions.
  • Review failures manually before changing candidate depth or fusion parameters.
  • Version retrieval policies so ranking changes can be compared and rolled back.