Why raw hybrid scores are difficult to combine

Dense retrieval ranks documents by proximity between embeddings. Sparse BM25 retrieval ranks documents using term-level evidence such as matching query terms, their rarity, and document-length effects. Both produce ordered results, but their numeric scores represent different models and are not a shared unit of relevance.

A fixed weighted sum of raw scores can appear to work for a small test set, then become unstable when queries vary in length, vocabulary, specificity, or embedding behavior. Score normalization can help in some systems, but it introduces assumptions about score distributions that must be monitored as the corpus and query mix evolve.

  • Use dense retrieval for semantic similarity and paraphrased intent.
  • Use BM25 for exact terms, identifiers, names, and uncommon vocabulary.
  • Treat each backend's ranking as reliable evidence even when its score scale differs.

Merge ranked lists with reciprocal rank fusion

Reciprocal rank fusion assigns each document a contribution based on its position in each result list, then adds those contributions together. For a document d, a common form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank from retrieval path i and k is a positive smoothing constant.

The smoothing constant reduces the gap between adjacent top positions and prevents a single rank-one placement from overwhelming all other evidence. The important property is not a universal value for k; it is that fusion relies on ordinal position rather than pretending dense and BM25 scores are directly interchangeable.

  • Request a candidate list from S3 Vectors for the dense query.
  • Request a candidate list from Quickwit BM25 for the sparse query.
  • Deduplicate by a stable document or chunk identifier.
  • Sum each document's reciprocal-rank contributions and sort descending.

Make fusion observable and easy to tune

Keep the source ranks and fusion contribution for every returned result. This makes retrieval debugging concrete: an engineer can see whether a result won because both paths agreed, because BM25 found an exact phrase, or because dense retrieval surfaced a semantically related passage.

Start with equal treatment of the two lists, then evaluate on representative queries before introducing path-specific weights. If one retrieval path is absent for a request or returns no candidates, fusion should still return the ranked results from the available path rather than failing the entire retrieval step.

  • Log document ID, dense rank, BM25 rank, and final fused rank.
  • Test exact-entity, paraphrase, multi-term, and ambiguous queries separately.
  • Choose candidate-list depth based on evaluation coverage, not intuition alone.
  • Version the fusion settings alongside query and indexing changes.