Why raw dense and BM25 scores should not be added directly

A dense-search score and a BM25 score are not automatically comparable. They may use different scales, distributions, and meanings. A score that is large in one retrieval system is not necessarily stronger evidence of relevance than a smaller score in another system.

This makes direct weighted addition fragile. It can appear to work for a small set of test queries, then change behavior when the corpus, embedding model, analyzer settings, or query vocabulary changes. Score normalization can help in some systems, but it introduces another calibration problem that must be monitored over time.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact terminology, product codes, names, and uncommon tokens.
  • Raw score ranges can differ even when both systems return relevant documents.
  • A fusion method based on rank avoids assuming score equivalence.

Fuse two candidate lists with reciprocal rank fusion

Reciprocal rank fusion, or RRF, combines ranked lists rather than their raw scores. For each document, add a contribution from every list in which it appears: 1 divided by a constant plus the document rank. Documents that rank well in either dense or sparse retrieval receive useful credit, while documents supported by both lists rise naturally.

For a query, request a candidate list from regional S3 Vectors and another from Quickwit BM25. Assign ranks starting at 1, group results by a stable document identifier, calculate the RRF total, sort descending, and return the highest-ranked documents. The constant, commonly written as k, reduces the difference between adjacent ranks and prevents the first position from overwhelming all other evidence.

  • RRF formula: score(d) = Σ 1 / (k + rank_i(d)).
  • Use the same stable document ID in dense and sparse indexes.
  • Retrieve more candidates than the final response size so fusion has meaningful overlap and alternatives.
  • Treat missing documents in a list as contributing zero from that list.

Make hybrid retrieval observable and testable

RRF is simple, but the surrounding retrieval pipeline still needs measurement. Build a query set that includes semantic questions, exact-match questions, mixed queries, abbreviations, and queries with expected zero results. Record the top dense candidates, top BM25 candidates, fused ranking, and the source lists that contributed to each final document.

This trace makes failures easier to diagnose. If an exact identifier is missed, inspect tokenization and BM25 indexing before changing fusion. If a paraphrased query misses relevant content, inspect embedding generation, chunking, and dense candidate depth. Fusion should combine useful candidate generators, not hide problems in either one.

  • Log document ID, rank, and retrieval source for each fused result.
  • Evaluate dense-only, BM25-only, and fused results on the same query set.
  • Version chunking, embedding, and indexing changes so ranking shifts are explainable.
  • Keep filters consistent across both retrieval paths before fusing results.