Why raw-score fusion is a trap

A dense retrieval system commonly returns a vector similarity or distance-derived value. BM25 returns a lexical relevance score based on term frequency, document statistics, and query terms. Even when both are presented as larger-is-better numbers, their scales are not inherently comparable.

A weighted expression such as dense_score + 0.3 × bm25_score can appear to work on a small test set, then drift as the corpus changes, analyzers are adjusted, embeddings are replaced, or a query contains unusually rare terms. The issue is not that either retriever is wrong; it is that the numeric outputs describe different ranking systems.

  • Dense retrieval helps with paraphrases and related concepts.
  • BM25 helps preserve exact matching for names, codes, and distinctive terminology.
  • Score ranges can shift independently as indexes and query distributions change.

Fuse ranked lists with reciprocal rank fusion

Run the same user query through both retrieval lanes, producing a ranked list from S3 Vectors and a ranked list from Quickwit BM25. Normalize document identity before merging: the same canonical document or chunk identifier must be used in both result sets.

For each document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank from retrieval lane i. A document absent from a list contributes nothing from that list. Sort documents by the final RRF score and pass the top merged candidates to the next stage, such as an application-side reranker or answer-generation context builder.

The constant k reduces the difference between nearby rank positions and prevents the first result in one list from overwhelming all other evidence. Treat k as an explicit configuration value, test it on representative queries, and keep it stable while evaluating other changes.

  • Fetch a bounded candidate list from each lane, such as the top N results.
  • Deduplicate by canonical chunk or document ID before returning results.
  • Use one-based ranks consistently in the fusion calculation.
  • Keep per-lane ranks in logs so merged results can be explained.

Make hybrid retrieval observable

RRF is simple enough to inspect, which makes it useful operationally. For every returned item, record whether it came from dense retrieval, sparse retrieval, or both, along with its rank in each lane and its final fused rank. This distinguishes a genuinely cross-signal result from one carried entirely by a single retriever.

Build an evaluation set from real query patterns rather than only polished examples. Include acronym-heavy queries, product names, error messages, natural-language questions, and ambiguous terms. Review not just whether a relevant item appears, but whether the candidate set contains the evidence needed by downstream ranking or generation.

When relevance changes, diagnose the lanes separately before changing fusion parameters. A sparse miss may point to tokenization or document text; a dense miss may point to chunking, embedding choice, or missing context. Fusion should combine useful candidate sets, not conceal a weak one.

  • Track overlap between dense and sparse top-N lists.
  • Sample queries where only one lane contributed the final top result.
  • Version chunking, indexing, and fusion configuration together.
  • Evaluate candidate recall before judging downstream answer quality.