Why raw score blending is often fragile

A dense-search score and a BM25 score are produced by different retrieval methods. Their numeric ranges, distributions, and meanings need not align. Adding them together or applying fixed weights can therefore make one retrieval path dominate simply because its scores are numerically larger.

This becomes especially brittle as indexes, embedding models, tokenization choices, or corpus composition change. A weight that appears sensible for one collection can behave differently for another, even when the query experience should remain consistent.

  • Dense retrieval can surface conceptually related documents when query wording differs from document wording.
  • BM25 can prioritize exact identifiers, product names, error codes, and uncommon terms.
  • Raw scores should not be treated as interchangeable unless they have been deliberately calibrated.
  • Rank positions are easier to combine because each retriever already expresses an ordering.

Apply reciprocal rank fusion to two result lists

Run the dense query against the S3 Vectors-backed path and the sparse query against the Quickwit BM25 path. Request a sufficiently deep candidate list from each retriever, then merge results by stable document identifier in the application layer.

For every document returned by either list, calculate an RRF score: 1 divided by k plus the document's rank in each list where it appears. Sum the contributions across lists, sort descending by the final value, and return the top results. The constant k reduces the influence of small rank differences near the top of a list.

  • Use one-based ranks: the first result has rank 1.
  • A document present in both lists receives two contributions and is naturally promoted.
  • Choose k as a configurable relevance parameter rather than treating it as a score-normalization constant.
  • Deduplicate by a canonical document or chunk ID before producing the final response.

Make fusion observable and safe to tune

Log the source ranks and fused rank for each returned document. These fields make it possible to see whether useful results are coming primarily from dense retrieval, BM25, or agreement between both. They also help diagnose cases where a lexical match or semantic match is unexpectedly displaced.

Evaluate changes using a representative query set with relevance judgments where possible. Inspect query groups separately, especially identifier-heavy queries, natural-language questions, short queries, and queries containing rare terminology. Tune candidate depth and k against retrieval quality rather than relying on a single global example.

  • Keep the original dense and BM25 ranks alongside the fused result for debugging.
  • Test what happens when a relevant item appears in only one retrieval path.
  • Use consistent document IDs across dense and sparse indexing pipelines.
  • Treat fusion configuration as versioned application behavior so ranking changes can be compared and rolled back.