Why score mixing is harder than it looks

Dense retrieval represents the meaning of a query and document as vectors, then retrieves nearby vectors. It can help when a user’s wording differs from the wording in the corpus. BM25 instead rewards lexical matches, including distinctive terms that may be important in identifiers, error messages, product names, and technical phrases.

It is tempting to add a dense score to a BM25 score and sort by the result. In practice, this assumes both values share a useful scale. That assumption is often fragile: a change in embedding model, index configuration, query type, or corpus composition can change the score distribution without changing what relevance means to users.

  • Dense search can retrieve semantic paraphrases.
  • BM25 can preserve exact-term precision.
  • Raw scores may not be directly comparable.
  • A rank-based method can provide a safer initial fusion strategy.

Fuse ranked lists with RRF

Run the same query through both retrieval paths: dense search against the S3 Vectors-backed index and sparse search through Quickwit BM25. Each path returns an ordered candidate list. RRF assigns every document a contribution based on its position in each list, then sums those contributions across lists.

For a document d, a common form is RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the influence of very high placements and makes the fusion less sensitive to small rank changes. A document appearing in both lists receives two contributions, while a document found by only one method can still remain competitive if it ranks highly there.

  • Retrieve a bounded top-N list from each search method.
  • Deduplicate candidates by a stable document or chunk identifier.
  • Use one-based ranks when calculating the RRF contribution.
  • Sort candidates by their summed RRF score before returning or reranking them.

Choose candidate depth and evaluate by query type

Candidate depth is a recall decision. If each retrieval path returns too few results, useful documents that rank moderately in one list cannot benefit from fusion. Start with a manageable depth that fits application latency and downstream processing, then evaluate whether relevant documents are reaching the fused candidate set.

Test RRF on a query set that reflects real retrieval behavior rather than only broad natural-language questions. Include acronym-heavy queries, exact identifiers, troubleshooting language, paraphrases, and multi-concept questions. Review failures separately: a poor result may come from chunking, metadata filtering, query construction, or missing corpus content rather than from the fusion rule itself.

  • Measure whether relevant items appear in the fused candidate set.
  • Compare dense-only, BM25-only, and fused rankings on the same queries.
  • Inspect queries where one retrieval method consistently dominates.
  • Keep fusion logic observable by logging ranks, source lists, and final positions.