Why raw-score blending is fragile

A dense retriever may return cosine similarity, dot-product similarity, or another vector-distance-derived score. BM25 returns a lexical relevance score influenced by term frequency, document length, and collection statistics. Even when both scores increase with relevance, their numeric ranges and distributions can differ substantially.

A weighted formula such as 0.5 × dense_score + 0.5 × bm25_score therefore embeds an assumption: that one point of dense score means roughly the same thing as one point of BM25 score. That assumption can change with embedding models, index settings, query language, corpus growth, and even the mix of documents in a result set.

  • Use raw-score blending only when scores have been deliberately calibrated and monitored.
  • Expect BM25 scores to vary with corpus and query-term characteristics.
  • Expect dense-score distributions to change when embeddings or similarity choices change.
  • Treat rank as a more portable signal when calibration is unavailable.

Fuse ranks with RRF

RRF assigns each document a contribution based on its position in each ranked list, then sums those contributions. For a document d, the common formulation is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i is the one-based position of d in retrieval list i and k is a positive constant.

Run a dense query against the S3 Vectors-backed path and a sparse query against the Quickwit BM25-backed path. Keep the top N identifiers from each result list, deduplicate identifiers, calculate RRF scores, and sort descending. A document returned by both systems is rewarded, while a document highly ranked by either system can still surface.

  • Start with the same candidate depth N for both retrievers, such as the top 50 or top 100.
  • Use a stable document identifier shared by dense and sparse indexes.
  • Choose a fixed k, often 60 as a conventional starting point, then evaluate it on your own relevance set.
  • Apply a deterministic secondary sort, such as document ID, when fused scores tie.

Make fusion observable and testable

RRF removes the need to normalize incomparable scores, but it does not remove the need for relevance evaluation. Build a small query set representing known user intents: exact identifiers, rare terms, paraphrases, multi-concept questions, and queries with ambiguous wording. For each query, record which documents should be considered useful.

Log the source ranks that produced each fused result. When a result is surprising, those ranks show whether the issue originated in dense retrieval, BM25 retrieval, document preparation, or fusion depth. This evidence is more actionable than inspecting a single blended score.

  • Track whether clicked or judged-relevant documents came from dense retrieval, sparse retrieval, or both.
  • Compare candidate depths before changing k; a missing document cannot be recovered by fusion.
  • Test exact-match queries separately because sparse retrieval may be especially important for them.
  • Re-evaluate after changing embeddings, analyzers, chunking rules, or document metadata.