Why raw-score blending is fragile

A tempting hybrid-search design is to request a dense result list and a BM25 result list, normalize both score columns, then add them with chosen weights. The problem is that dense similarity and BM25 relevance scores are produced by different ranking functions. Their numeric ranges, distributions, and sensitivity to query length can differ substantially.

Even a normalization method that works on one query set can become unstable when content, embedding models, or query patterns change. This can turn weight tuning into an ongoing maintenance task. Rank fusion avoids treating the two score values as directly comparable: it works from each retriever’s ordered list instead.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact vocabulary, product names, codes, and rare terms.
  • A document’s position in a result list is often easier to combine than unrelated raw scores.

Fuse dense and sparse lists with RRF

Reciprocal rank fusion (RRF) assigns each document a contribution based on its rank in each list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i is the one-based rank from retrieval list i and k is a positive constant. Add contributions from the dense and BM25 lists, then sort documents by the combined value.

Use a stable document identifier as the join key. A document returned by both searches receives two contributions; a document returned by only one search can still rank well if it appears near the top of that list. RRF therefore preserves useful candidates that a single retriever might miss.

  • Retrieve a bounded candidate set from regional S3 Vectors for dense search.
  • Retrieve a bounded candidate set from Quickwit BM25 for sparse search.
  • Deduplicate candidates by document ID before presenting results.
  • Start with the same candidate depth for both lists, then inspect misses before changing it.

Tune the retrieval pipeline with judged queries

Treat fusion as a retrieval-stage decision, not a substitute for evaluation. Assemble a small set of representative queries and record which documents are useful for each. Include semantic questions, exact-title searches, acronym-heavy queries, identifier lookups, and ambiguous terms. These categories reveal where dense and sparse retrieval complement each other.

Evaluate the fused ranking alongside dense-only and BM25-only baselines. Review not just aggregate relevance but also failure modes: missing exact matches, semantically plausible but wrong results, duplicate chunks, and results dominated by one source. If a result set will feed a generation step, keep enough provenance to inspect why a document entered the final context.

  • Log the source ranks that contributed to every fused result.
  • Keep document IDs and chunk boundaries consistent across both indexes.
  • Change one variable at a time: candidate depth, k, or chunking strategy.
  • Use production query samples only after removing or protecting sensitive data.