Why raw-score blending is fragile

In a Talqora retrieval design, dense candidates come from regional S3 Vectors while sparse candidates come from Quickwit BM25. Each system returns a ranking signal with its own meaning, distribution, and query-dependent behavior.

For one query, the top dense result may be far ahead of the rest; for another, dense scores may be tightly grouped. BM25 has similar variation based on term frequency, document length, and query specificity. A formula such as dense_score plus BM25_score can therefore give one retriever accidental dominance.

  • Do not assume a dense similarity score and a BM25 score are numerically comparable.
  • Avoid selecting fixed weights solely because they work for a small set of example queries.
  • Treat ranking and score calibration as separate engineering concerns.

Fuse ranks with RRF

RRF assigns each document a contribution based on its position in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across the lists where d appears. The constant k reduces the difference between nearby top ranks and is commonly chosen as a positive tuning parameter.

First retrieve a bounded candidate list from dense search and another from BM25. Deduplicate by a stable document or chunk identifier, calculate the fused score, and sort descending. A document found by both methods gains two contributions, while a strong single-method result can still remain competitive.

  • Run dense retrieval against S3 Vectors and sparse retrieval against Quickwit BM25 for the same query.
  • Keep result-list depth explicit, such as a configurable top-N per retriever.
  • Deduplicate before returning results so a chunk has one final rank.
  • Store per-retriever rank and fused score for debugging.

Evaluate the fusion path, not only the retrievers

Build a small evaluation set from representative requests, including semantic questions, exact product names, acronyms, codes, and recently introduced terminology. Record whether a useful document appears in the dense list, the BM25 list, and the final fused list.

RRF is not a substitute for relevance evaluation. Use the observations to adjust candidate depth, the fusion constant, chunking choices, and metadata filters. If later you have reliable judged data, calibrated score-based blending or a learned reranker may be worth testing—but they should be compared against a simple rank-fusion baseline.

  • Inspect queries where dense and sparse retrieval disagree.
  • Measure recall at the candidate stage before judging final answer quality.
  • Log retrieval configuration alongside results to make ranking changes reproducible.
  • Apply the same authorization and metadata filters to every retrieval path.