Why raw-score blending is fragile

A dense-search score reflects a vector similarity calculation. A BM25 score is based on term statistics and document-length-aware lexical matching. Even if both systems return numeric scores, a score of 0.8 from one retriever does not inherently have the same relevance meaning as 0.8 from the other.

Directly weighting and summing those scores creates a calibration problem. The resulting rank can change because one retriever emits values over a wider range, not because its results are more useful for the query. This can be difficult to notice until exact-match queries or vocabulary-mismatch queries behave unexpectedly.

  • Avoid assuming dense and BM25 score ranges are comparable.
  • Do not normalize scores globally without checking query-level behavior.
  • Treat rank fusion as a strong baseline before adding score calibration.

Fuse two ranked lists with RRF

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i is the one-based position returned by retriever i and k is a positive constant selected by the application.

Retrieve a bounded candidate list from S3 Vectors and another from Quickwit BM25, then merge documents by a stable document ID. Add the RRF contribution for every list in which a document appears, sort by the fused score, and return the top results. Documents found by both retrieval methods naturally receive support from both lists.

  • Use the same canonical document ID in dense and sparse indexes.
  • Request enough candidates from each retriever to allow meaningful overlap and recovery.
  • Choose k deliberately and keep it fixed while establishing a baseline.
  • Preserve source ranks for debugging and evaluation.

Make fusion observable before tuning it

RRF is useful partly because it is easy to inspect. For each returned result, record whether it originated from dense retrieval, BM25, or both, along with its rank in each source list. This makes it possible to understand whether a result won through lexical precision, semantic matching, or agreement between the two.

Evaluate with a query set that represents real retrieval intent. Include exact IDs and product names, natural-language questions, abbreviated queries, and queries whose wording differs from the target content. Review failures by query class before changing candidate depth or the fusion constant.

  • Log fused rank, per-retriever rank, and document ID.
  • Test lexical and semantic query classes separately.
  • Inspect cases where one retriever displaces a clearly relevant result.
  • Only introduce learned or score-based blending after a rank-fusion baseline is understood.