Why raw-score addition is fragile

A dense-search score reflects the similarity function and embedding representation used for a vector query. A BM25 score reflects lexical term matching, term frequency, document frequency, and document-length effects. Even when both searches return a numeric score, those numbers do not inherently mean the same thing.

Adding scores directly can make one retriever dominate for reasons unrelated to result quality. A change in embedding model, index configuration, corpus composition, or query wording may shift score distributions. The result is a fusion rule that is difficult to reason about and easy to destabilize.

  • Do not assume a dense similarity score and a BM25 score have equivalent ranges or interpretation.
  • Avoid setting a single global weight before inspecting how both result lists behave across query types.
  • Treat score calibration as a separate problem from candidate generation.

Use reciprocal rank fusion for a robust first pass

Reciprocal rank fusion, commonly abbreviated as RRF, combines ranked lists instead of combining their original scores. For each document, add a contribution from every list in which it appears: 1 divided by k plus its rank. The constant k reduces the difference between adjacent high ranks and prevents a single first-place result from overwhelming the combined list.

In a Talqora-backed retrieval flow, an application can request a top-N dense list from regional S3 Vectors and a top-N sparse list from Quickwit BM25. It can then deduplicate document IDs, calculate an RRF value for each candidate, and sort by that value. This logic lives cleanly in the application layer because it only requires ranked outputs.

  • Run dense and sparse retrieval with the same query intent, while allowing each path to use its appropriate query representation.
  • Choose a candidate depth large enough to capture useful overlap and complementary results.
  • Assign ranks starting at 1, then compute contributions consistently for every list.
  • Keep document identifiers stable across dense and sparse indexes so duplicates can be merged.

Evaluate overlap and failure modes before tuning

Rank fusion is not a substitute for evaluation. Build a small query set that represents the language users actually enter: natural-language questions, exact product names, abbreviations, identifiers, misspellings if relevant, and multi-concept requests. Review whether the fused list preserves strong exact matches while adding semantically related material.

It is also useful to log which retrieval path contributed each fused candidate. A document found by both paths is a different signal from one found only by BM25 or only by dense search. That visibility helps diagnose whether a query needs better chunking, metadata filtering, query rewriting, or a different candidate depth.

  • Compare dense-only, BM25-only, and fused candidate lists for the same evaluation queries.
  • Inspect queries where one retriever contributes no useful candidates.
  • Track the source list or lists that contributed each final candidate.
  • Tune candidate depth and the RRF constant only after reviewing representative results.