Why raw score blending is brittle

Dense search and BM25 produce scores with different meanings. A dense score typically reflects the relationship between embedded query and document vectors, while BM25 is based on lexical term statistics. Even when each system ranks relevant documents well, their numeric outputs are not automatically comparable.

A weighted formula such as dense_score × 0.7 plus bm25_score × 0.3 requires stable score distributions and careful calibration. Those distributions can change with corpus composition, query length, analyzer configuration, embedding model changes, or retrieval parameters. Rank-based fusion avoids treating these unlike values as though they share a common unit.

  • Use each retriever’s ordering as the primary signal.
  • Keep dense and sparse retrieval requests independently configurable.
  • Log ranks and source lists before attempting score-level tuning.

Apply reciprocal rank fusion

RRF assigns a contribution to a document based on its position in each result list. For a document d, the fused score is the sum of 1 divided by k plus rank(d) across the lists in which d appears. The constant k reduces the impact of small rank differences near the top of a list.

For example, an application can request a candidate list from regional S3 Vectors and a candidate list from Quickwit BM25, deduplicate document identifiers, calculate the RRF score for every candidate, and sort descending. A document appearing in both lists is reinforced; one appearing highly in only one list can still remain competitive.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first result has rank 1.
  • Choose a fixed k initially and evaluate it against representative queries.
  • Treat a missing document from a list as contributing zero, not as a very large rank.

Make fusion operationally useful

Fetch enough candidates from each retriever to create a meaningful union, then return only the final number needed by the user interface or downstream system. If either candidate list is too shallow, a document that would have benefited from overlap may never enter the fusion step.

Evaluation should include query classes rather than only an aggregate metric. Exact identifiers, product names, error codes, and quoted phrases often test lexical retrieval differently from conceptual or paraphrased questions. Review whether fusion improves both categories, and inspect failures where one retriever’s distinctive result was pushed down.

  • Use stable document IDs across dense and sparse indexes for deduplication.
  • Record per-document dense rank, BM25 rank, and fused rank for debugging.
  • Test exact-match and semantic query sets separately.
  • Revisit candidate depth and k when the corpus, embedding model, or BM25 configuration changes.