Why raw-score addition is fragile

Talqora uses regional S3 Vectors for dense search and Quickwit BM25 for sparse search. These retrieval paths can return useful results for the same query, but their scores do not necessarily share a common meaning or range.

A dense similarity score and a BM25 score are produced by different methods. Even if both lists are sorted correctly on their own, adding the values together can let one retrieval path dominate simply because its numeric scale is larger. Normalizing scores can help, but it introduces choices that need monitoring as content and query traffic change.

  • Dense retrieval helps with paraphrases, concepts, and semantically related language.
  • BM25 helps with exact names, error codes, product terms, and uncommon tokens.
  • A score value is meaningful within its own ranking system; it is not automatically comparable across systems.

Fuse ranked lists with reciprocal rank fusion

Reciprocal rank fusion (RRF) combines result lists using positions rather than raw scores. Retrieve a candidate list from dense search and another from BM25, then assign each document a fused value based on where it appears in each list.

For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), across the lists where d appears. The constant k softens the difference between adjacent positions. A document that ranks well in both lists rises naturally, while a strong result unique to one method can still remain competitive.

  • Use one-based ranks: the first result has rank 1.
  • Deduplicate by a stable document or chunk identifier before returning results.
  • Choose a fixed k initially and evaluate it with representative queries rather than tuning against a single example.
  • Keep the dense and sparse candidate depths large enough that useful overlap can occur.

Make fusion observable before making it sophisticated

Start by logging which retrieval path contributed each result, its dense rank, its BM25 rank, and its final fused rank. This makes hybrid behavior inspectable: a result may win because both methods agree, because BM25 found an exact match, or because dense retrieval found a semantic alternative.

Evaluate with a small, maintained query set that reflects real retrieval needs. Include exact-lookup queries, terminology-heavy queries, paraphrases, and ambiguous requests. When a result is wrong, inspect the separate candidate lists before changing fusion logic; the issue may be chunking, metadata filtering, embedding choice, or missing source content rather than RRF itself.

  • Track overlap between dense and sparse candidate lists.
  • Review zero-result and low-quality queries separately from ranking disagreements.
  • Apply the same filters and access controls to both retrieval paths before fusion.
  • Treat RRF as a transparent baseline; add more complex reranking only when evaluation identifies a specific gap.