Why raw-score blending is brittle

A dense-search score reflects the relationship between a query embedding and a document embedding. A BM25 score reflects term frequency, document frequency, field length, and query-term matches. Even if both systems return a number called a score, the numbers do not represent the same quantity.

A fixed weighted formula, such as 0.6 times a dense score plus 0.4 times a BM25 score, can therefore behave unpredictably. A change to embedding generation, corpus composition, BM25 field configuration, or query length may change score distributions without changing what users consider relevant.

  • Do not assume a dense score of 0.8 is inherently stronger than a BM25 score of 8.
  • Avoid tuning weights from a small set of memorable queries alone.
  • Treat each retrieval system first as a ranked-candidate generator.

Fuse ranks with RRF

RRF assigns each document a contribution based on its position in each result list. For a document d, sum 1 divided by k plus its rank in every list where it appears. The constant k reduces the impact of small rank differences near the top of a list.

For example, a document ranked third by dense retrieval and tenth by BM25 receives contributions from both lists. A document that appears only in one list can still rank well, but documents supported by both lexical and semantic evidence tend to rise.

  • Run the dense query against the S3 Vectors-backed candidate set.
  • Run the sparse query through the Quickwit BM25 candidate set.
  • Use one-based ranks and deduplicate documents by a stable document identifier.
  • Sort candidates by their summed RRF score before any later reranking step.

Make fusion observable and query-aware

Log more than the final ranking. For each returned document, record its dense rank, BM25 rank, fused score, and which lists contained it. This makes it possible to distinguish a failure to retrieve from a failure to fuse or present the best candidate.

Start with a consistent candidate depth for both retrieval paths, then evaluate on a small, representative set of real queries. Include queries dominated by exact entities, broad conceptual questions, short ambiguous requests, and terminology that differs between the query and the target document.

  • Inspect overlap between dense and BM25 result lists; low overlap is not automatically a problem.
  • Track whether relevant documents appear in either candidate list before judging fusion.
  • Review exact-match queries separately from semantic paraphrase queries.
  • Keep fusion logic deterministic so ranking changes can be traced to retrieval inputs or configuration changes.