Why raw-score fusion is fragile

A dense retriever and a BM25 retriever produce scores for different reasons. A vector similarity score reflects distance or similarity in embedding space. A BM25 score reflects term frequency, document frequency, and field-length effects. Even when both systems return a number called a score, those numbers are not automatically comparable.

Min-max scaling, z-scores, and hand-tuned weighted sums can work in controlled settings, but they introduce operational questions: which candidate set defines the normalization range, how should zero-result queries behave, and when should weights change after an embedding or analyzer update? Rank-based fusion avoids most of those assumptions.

  • Use dense retrieval for semantic recall.
  • Use BM25 for exact language, codes, names, and uncommon terms.
  • Treat each retriever's rank order as more portable than its raw score.
  • Keep original scores in logs for diagnosis, even if fusion does not use them.

Apply Reciprocal Rank Fusion to a shared candidate set

RRF assigns each document a contribution based on its position in each result list. For a document d, a common form is: RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs over retrieval lists in which d appears, rank starts at 1, and k is a smoothing constant. Documents appearing near the top of both lists receive a strong combined signal.

The important implementation detail is identity. Dense and sparse indexes must use the same stable document or chunk identifier so results can be joined deterministically. If a source document is split into chunks, use a chunk-level ID for fusion and retain a parent document ID for later grouping or display.

  • Retrieve a bounded top-K list from regional S3 Vectors.
  • Retrieve a bounded top-K list from Quickwit BM25.
  • Deduplicate by a stable canonical chunk ID before presenting results.
  • Choose and record one k value; change it deliberately rather than per request.

Make the fusion layer observable and testable

Because RRF is simple, it is well suited to an API-layer retrieval pipeline. Log the query, retrieval source, rank, canonical ID, fused rank, and filter context. These records make it possible to answer whether a result was promoted because both retrievers found it or because it ranked highly in only one.

Evaluate with a small set of representative queries before expanding complexity. Include acronym-heavy queries, exact part numbers, paraphrases, multi-word natural-language questions, and queries that should return nothing. The objective is not to prove that one retriever always wins; it is to verify that the combined ranking preserves useful behavior from each retrieval path.

  • Track overlap between dense and BM25 candidate lists.
  • Inspect queries where only one retrieval path contributes top results.
  • Version embeddings, chunking rules, and BM25 indexing settings separately.
  • Add reranking only after the fused candidate set is stable and inspectable.