Why raw score blending is fragile

It is tempting to retrieve documents from both systems and calculate a weighted sum of their scores. In practice, a dense-search similarity score and a BM25 score are produced by different ranking functions and have different ranges, distributions, and meanings.

A weight that works for one corpus, query type, or indexing configuration may work poorly after content changes. Score normalization can help, but it introduces another set of assumptions that must be measured and maintained.

  • BM25 scores depend on term frequency, document frequency, and document length.
  • Dense similarity depends on the embedding model, vector normalization, and similarity metric.
  • Identical numeric values from the two retrievers do not imply identical relevance.

Fuse ranks with reciprocal rank fusion

Reciprocal rank fusion, or RRF, combines ordered result lists rather than comparing raw scores. For each document, add a contribution from every list in which it appears: 1 divided by k plus the document’s rank in that list.

The constant k reduces the influence of small rank differences near the top of a list. The resulting fused score is useful only for ordering the combined candidates; it does not need to be interpreted as a universal relevance probability.

  • Retrieve a candidate list from S3 Vectors for the dense query.
  • Retrieve a candidate list from Quickwit BM25 for the lexical query.
  • Assign each document an RRF score: sum(1 / (k + rank)).
  • Deduplicate by stable document ID, sort by fused score, and return the top results.

Make fusion observable and easy to tune

Start with the same candidate depth for both retrieval paths and a fixed k value. Keep the dense rank, sparse rank, fused rank, and retrieval source for every returned document in diagnostic output. This makes it possible to identify whether a result was supported by semantic matching, lexical matching, or both.

Evaluate changes using a representative query set that includes exact identifiers, short natural-language questions, domain terminology, and ambiguous queries. Tune candidate depth and k only against judged outcomes, not against a handful of memorable searches.

  • Use stable IDs shared by the dense and sparse indexes.
  • Log missing results separately from poor ordering.
  • Inspect queries where one retrieval path contributes most of the final results.
  • Re-run evaluations after embedding, analyzer, or corpus changes.