Why raw-score blending is fragile

A dense retrieval score and a BM25 score are produced by different ranking models. Their numeric ranges, distributions, and sensitivity to query length can differ substantially. A score of 0.8 from one system is not inherently comparable to a score of 0.8, 8, or 80 from another.

A simple weighted sum can therefore behave unpredictably. A small change in tokenization, corpus composition, embedding model, or retrieval configuration may alter one score distribution enough to dominate the combined ranking, even if the underlying result quality has not changed.

  • Do not assume dense and BM25 scores share a common scale.
  • Avoid choosing weights solely because a score range looks larger or smaller.
  • Treat score normalization as a model decision that requires evaluation data.

Fuse result ranks with Reciprocal Rank Fusion

Reciprocal Rank Fusion, often abbreviated RRF, combines ranked lists rather than their raw scores. For each document, add a contribution from every list in which it appears: 1 divided by k plus the document's rank. The constant k reduces the impact of small differences near the very top of a list.

For a query, retrieve a candidate set from regional S3 Vectors and another from Quickwit BM25. Deduplicate documents by a stable document identifier, calculate the RRF total for each candidate, then sort by that total. A document that ranks well in both systems rises naturally, while a strong result from either retriever can still remain visible.

  • Retrieve the same candidate depth from both systems as a starting point.
  • Use one stable ID for deduplication across dense and sparse results.
  • Choose and record a fixed k value before evaluating changes.
  • Keep each source rank available for debugging the fused result.

Evaluate the retrieval behavior, not just the formula

Rank fusion is simple, but it should still be tested against representative queries. Build a small evaluation set that includes exact-title searches, identifiers, abbreviations, multi-word concepts, and queries with vocabulary that differs from the target documents. These categories reveal where sparse and dense retrieval complement each other.

Inspect failures at the document level. If an exact part number is missing, the sparse candidate depth or indexing pipeline may deserve attention. If a paraphrased question retrieves only lexical matches, the dense candidate set may be too shallow or the embedding representation may need review. Fusion cannot recover documents that neither retriever returns.

  • Measure whether a relevant document enters the fused top results.
  • Compare dense-only, sparse-only, and fused rankings on the same queries.
  • Log source ranks and final rank for difficult queries.
  • Adjust candidate depth before making fusion logic more complex.