Why raw score blending is fragile

A dense-search score and a BM25 score are produced by different retrieval models and have different ranges, distributions, and meanings. A value that looks large in one system is not automatically more relevant than a smaller-looking value from the other. Score ranges can also shift as embeddings, analyzers, document lengths, or index settings change.

Normalizing scores can be useful in a controlled evaluation setup, but it introduces assumptions about score distributions and calibration. For a first hybrid retrieval implementation, rank-based fusion is usually easier to reason about because it asks a simpler question: did a document appear near the top of either candidate list?

  • Do not add dense and BM25 scores unless they have been deliberately calibrated.
  • Preserve document IDs consistently across the dense and sparse indexes.
  • Request enough candidates from each retrieval path to make fusion meaningful.
  • Treat ranking changes as an evaluation question, not as proof that one score scale is better.

Fuse ranked lists with reciprocal rank fusion

Reciprocal rank fusion assigns each document a contribution based on its position in each result list. For a document d, the fused score is the sum of 1 divided by k plus its rank for every list in which it appears. The constant k reduces the difference between adjacent top ranks and prevents a single rank-one result from overwhelming all other evidence.

In a Talqora-oriented retrieval flow, an application can use dense candidates backed by regional S3 Vectors and sparse candidates backed by Quickwit BM25. It then joins the returned document IDs in application logic, computes the RRF score, sorts the union, and returns the top fused results. This keeps each backend responsible for what it does best while making the combination explicit and testable.

  • Use one-based ranks: rank 1 is the first result in a list.
  • A common starting formula is: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Deduplicate by stable document ID before producing the final ranking.
  • Keep the source ranks with each result so relevance investigations remain possible.

Build an observable query path

Hybrid retrieval should be inspectable at the query level. Store or log the dense rank, sparse rank, fused rank, and the retrieval path that contributed to each returned document. When a result surprises a user or evaluator, this record shows whether the issue began in candidate generation, ID mapping, fusion, or later reranking.

Evaluate with queries that represent the reasons for using two retrieval modes. Include exact strings such as ticket IDs and error messages, semantic paraphrases, acronym-heavy requests, and mixed queries containing both a concept and a precise term. Compare dense-only, sparse-only, and fused result lists against the same relevance judgments.

  • Use a fixed test set before changing candidate depths or the RRF constant.
  • Inspect queries where only one retrieval path found the relevant document.
  • Check that updates and deletes leave no stale ID mappings between indexes.
  • Version query construction and fusion settings alongside relevance evaluations.