Why raw dense and BM25 scores are a poor common currency

Dense retrieval ranks documents by a vector-similarity function, while BM25 ranks them using term statistics and document-length normalization. Even when both systems return a numeric score, the meaning, range, and distribution of those values can differ.

A weighted expression such as 0.5 × dense_score + 0.5 × bm25_score therefore has an unstated assumption: that a one-unit change means roughly the same thing in both rankings. That assumption can break when embedding models, corpus composition, analyzers, or query shapes change.

  • A query containing an exact ticket ID may be best handled by BM25.
  • A paraphrased natural-language question may be best handled by dense retrieval.
  • Score ranges can shift without any change to application-side weights.
  • Min-max normalization can also be unstable when the top result set changes.

Fuse ranks, not scores, with RRF

Reciprocal Rank Fusion assigns each document a contribution based on its position in each result list. For a document d, add 1 divided by k plus its rank for every list in which it appears. The final score is the sum of those contributions, and documents are sorted by that sum.

In a Talqora retrieval flow, request a candidate list from regional S3 Vectors for dense search and another from Quickwit BM25 for sparse search. Use the document ID as the join key, calculate RRF in the application or retrieval orchestration layer, then return the highest fused results.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first item in a list has rank 1.
  • Choose a candidate depth for each retriever before fusion, such as the top N results.
  • Start with the same k for both lists; treat it as a ranking-shape parameter, not a score calibration value.

Implement the merge defensively and evaluate query slices

Deduplicate by a stable canonical ID before scoring. A document returned by both retrieval paths should receive two RRF contributions; a document returned by only one path should remain eligible. Keep the source ranks during development so that unexpected results can be inspected without guessing which retriever introduced them.

Evaluate the fused ranking against representative query categories rather than relying on a single aggregate judgment. Include exact identifiers, short keyword queries, long natural-language requests, uncommon terminology, and queries with known relevant documents. This makes it easier to spot whether candidate depth or an upstream retrieval configuration is limiting recall.

  • Log document ID, dense rank, BM25 rank, and fused score for sampled requests.
  • Make filtering rules consistent across both candidate searches when the product requires the same access or metadata constraints.
  • Test candidate depths separately from final page size; fusion cannot promote a document that neither retriever returned.
  • If later ranking is needed, use the fused set as a compact candidate pool for a separate reranking step.