Why raw-score blending is fragile

A dense retrieval score and a BM25 score are not inherently comparable. Their ranges, distributions, and meanings depend on the embedding model, index configuration, corpus, query length, and scoring implementation. Adding them together can make one retriever dominate simply because its numbers are larger.

Normalization can help, but it introduces another moving part. A min-max range calculated for one query may not represent another, and corpus changes can alter score distributions over time. For many retrieval systems, ranking position is a more stable signal than the numeric score emitted by either retriever.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact vocabulary, part numbers, error codes, and named entities.
  • Score magnitudes should not be assumed to have the same meaning across retrievers.

Use reciprocal rank fusion as the merge step

Reciprocal rank fusion, or RRF, combines result lists by awarding each document points based on its position in each list. For a document d, the fused score is the sum of 1 divided by k plus its rank for every list where it appears: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the impact of small rank differences near the top of a list.

In a Talqora-oriented architecture, dense candidates can come from regional S3 Vectors and lexical candidates can come from Quickwit BM25. Retrieve a bounded candidate list from each source, join entries by a shared document or chunk identifier, calculate RRF, and return the highest fused ranks. This approach uses each system's ordering rather than attempting to calibrate their native scores.

  • Use the same stable ID for the corresponding dense vector and BM25 document.
  • Request a candidate depth larger than the final page size, such as a few dozen results from each retriever.
  • Choose one RRF constant and evaluate it against representative queries before changing it.
  • Treat a document absent from a list as contributing zero from that list.

Make fusion observable and easy to tune

Store enough retrieval metadata to explain a fused result. At minimum, record the dense rank, BM25 rank, fused score, and the retrieval path or paths that returned the item. This makes it possible to distinguish a result supported by both retrievers from one that was promoted by a single strong signal.

Evaluate hybrid behavior with a query set that includes both semantic and lexical cases. Include natural-language questions, terminology-heavy queries, exact IDs, quoted phrases, and queries with spelling or wording variation. The goal is not for hybrid retrieval to win every query; it is to reduce blind spots created by relying on only one retrieval method.

  • Keep document chunking and IDs consistent across dense and sparse indexes.
  • Log the top candidates before and after fusion for debugging.
  • Review failures by query type, not only by an aggregate relevance metric.
  • Add later ranking stages only after the candidate-generation path is understandable.