Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different models and scoring rules. Even when both are larger for better matches, their numeric ranges, distributions, and sensitivity to query length can differ. Adding them directly creates an implicit claim that one unit of dense score means the same thing as one unit of BM25 score.

That assumption is often unstable. A score weighting that appears useful for a small evaluation set may change as documents are added, embeddings are regenerated, analyzers change, or query traffic shifts. Before tuning score weights, establish a reliable baseline that does not depend on cross-system score comparability.

  • Dense retrieval favors semantic similarity.
  • BM25 rewards term overlap and term specificity.
  • Raw scores may have different scales and meanings.
  • Rank positions are easier to compare across result lists.

Fuse two ranked lists with RRF

Run dense retrieval against Talqora's regional S3 Vectors-backed dense search and sparse retrieval against its Quickwit BM25-backed search. Request a sufficiently deep candidate list from each path, then combine the two lists in the application. For every document appearing in either list, add an RRF contribution based on its position.

The standard formula is RRF(d) = sum of 1 divided by (k plus rank_i(d)), where rank_i(d) is document d's one-based rank in list i and k is a positive constant. A document found by both methods accumulates contributions. A document ranked highly by one method remains competitive even when it does not appear in the other list.

  • Use one-based ranks: first result has rank 1.
  • Choose the same candidate depth for initial experiments.
  • Treat absent documents as contributing zero from that list.
  • Sort documents by descending fused score and return the top results.

Make fusion observable before making it complex

Log the dense rank, sparse rank, fused rank, and source identifiers for a sample of queries. These fields make it possible to see whether a result won because both retrieval methods agreed or because one method supplied a strong unique match. They also make regressions diagnosable when an index, embedding pipeline, or query formulation changes.

Evaluate using representative queries rather than a single aggregate number alone. Include exact identifiers, acronyms, paraphrased questions, broad topical searches, and queries with ambiguous terminology. If the baseline exposes a consistent weakness, then consider extensions such as weighted RRF, metadata filtering before fusion, or a later reranking stage. Each added component should have a specific failure mode it is intended to address.

  • Record per-source ranks alongside the final rank.
  • Inspect queries where dense and sparse results disagree.
  • Test candidate depth separately from the RRF constant.
  • Keep a fixed evaluation set when changing retrieval logic.