Why raw-score blending is fragile

A dense search score is produced from an embedding similarity calculation, while BM25 ranks documents from term frequency, document frequency, and field-length effects. Even when both systems return numeric scores, a score of 0.8 from one system has no inherent relationship to a score of 0.8 from the other.

Normalizing scores per query can help in some designs, but it introduces additional assumptions. A small change in the candidate set, query wording, or score distribution can change the normalization outcome. Rank-based fusion avoids requiring a shared score meaning.

  • Dense retrieval can surface paraphrases and conceptually related content.
  • BM25 can preserve exact identifiers, error codes, names, and uncommon terms.
  • Raw scores should be treated as system-specific unless they have been explicitly calibrated.

Fuse ranks with reciprocal rank fusion

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is its one-based position in result list i and k is a positive constant. Documents that rank well in either list, and especially in both, rise toward the top.

For Talqora Vector, request a candidate list from regional S3 Vectors and a candidate list from Quickwit BM25, then merge by a stable document identifier. Sum each document's RRF contributions and sort descending. The fusion layer does not need to inspect or compare the underlying raw scores.

  • Use the same canonical document ID in dense and sparse indexes.
  • Choose a fixed k and record it as retrieval configuration.
  • Fetch more candidates from each retriever than the final number of results returned.
  • Apply a deterministic tie-breaker, such as document ID, after the fused score.

Make the fusion path observable

A fused ranking is easier to tune when each result retains provenance. Store whether a document appeared in dense retrieval, sparse retrieval, or both, along with its rank and RRF contribution from each source. This makes unexpected rankings explainable during debugging.

Review representative query groups rather than relying on a single query type. Queries containing product names, IDs, or quoted phrases often reveal sparse-retrieval behavior, while natural-language questions can reveal dense-retrieval behavior. The goal is not to make both systems agree on every query; it is to use their complementary signals reliably.

  • Log dense rank, sparse rank, fused score, and source presence per returned document.
  • Track candidate counts before and after deduplication.
  • Test exact-match, semantic, mixed-intent, and no-result queries.
  • Version retrieval settings so ranking changes can be traced to a configuration change.