Why raw dense and sparse scores should not be added directly

A dense retrieval score and a BM25 score are produced by different ranking functions. Their ranges, distributions, and behavior can change with the query, corpus, indexing configuration, or retrieval implementation. A score of 0.8 from one system does not inherently represent the same evidence as a score of 0.8 from another.

Adding raw scores can therefore make one retriever dominate for accidental numerical reasons rather than because it found better documents. Score normalization can help, but it introduces another calibration problem: the normalization strategy itself must be selected, monitored, and revised as data changes.

  • Dense retrieval can surface semantic matches that do not share query terms.
  • BM25 can strongly reward exact terminology, identifiers, names, and rare tokens.
  • Raw score scales are not a shared relevance scale.
  • Rank-based fusion avoids requiring one.

Fuse result ranks with reciprocal rank fusion

Run dense retrieval and BM25 retrieval independently for the same query, then retain an ordered candidate list from each. For every document appearing in either list, calculate an RRF contribution from each ranking: 1 divided by k plus the document rank. Sum the contributions across lists, then sort documents by the resulting total.

The constant k reduces the difference between nearby positions and prevents the first few ranks from overwhelming all other evidence. Use the same value consistently at first, then evaluate changes against a labeled query set or structured relevance review. The key operational advantage is that RRF only requires rank positions, not calibrated scores.

  • RRF(document) = Σ 1 / (k + rank_i(document))
  • Assign rank 1 to the first result in each retriever's list.
  • A document absent from a list contributes zero from that list.
  • Deduplicate by a stable document or chunk identifier before returning results.

Choose candidate depth and inspect disagreement

Fusion can only promote documents that one of the retrievers returned. Request enough candidates from both the S3 Vectors-backed dense path and the Quickwit BM25 path to give the merger useful overlap and complementary coverage. Candidate depth should be treated as a retrieval parameter, not merely a pagination setting.

Log the fused rank, each source rank, and whether a result came from dense retrieval, BM25, or both. These fields make failures easier to diagnose. If a relevant result appears only in sparse search, inspect terminology and embedding coverage; if it appears only in dense search, inspect whether exact terms, metadata, or tokenization are limiting the sparse path.

  • Start with equal candidate depths for dense and sparse retrieval.
  • Measure overlap between the two lists for representative queries.
  • Keep source ranks in debugging and evaluation output.
  • Evaluate final fused ordering, not either retriever in isolation.