Why raw dense and BM25 scores should not be added

Dense retrieval ranks documents by vector similarity, while BM25 ranks them from token-level term matching and corpus statistics. Even when both systems return a numeric score, the scale, distribution, and meaning of those scores are different.

Adding raw scores can make one retrieval path dominate merely because its scores have a larger numeric range. Normalizing scores can help in some systems, but normalization depends on the result set and can behave differently as the corpus, query mix, or retrieval depth changes.

  • Dense retrieval is useful when query and document wording differs but meaning is related.
  • BM25 is useful when exact terms, identifiers, names, and rare vocabulary matter.
  • Rank positions are easier to compare across retrieval methods than raw scores.

Apply Reciprocal Rank Fusion to two result lists

Run the same user query through the dense path backed by regional S3 Vectors and the sparse path backed by Quickwit BM25. Keep a bounded ranked list from each path, such as the top N document IDs, and then aggregate results by document ID.

For each document, add a contribution based on its rank in every list where it appears: 1 divided by k plus the rank. The constant k reduces the impact of very small rank differences near the top while still rewarding documents that rank well in either list.

  • RRF score: score(document) = Σ 1 / (k + rank).
  • Use ranks starting at 1, not 0.
  • Choose the same retrieval depth for both lists initially to simplify evaluation.
  • Deduplicate by a stable document or chunk identifier before returning the final ranking.

Make fusion observable before tuning it

Log the dense rank, BM25 rank, and fused rank for returned items. These fields make it possible to see whether an item won because of semantic similarity, exact matching, or agreement between both paths. They also help diagnose cases where a document appears in only one candidate list.

Evaluate the fused list with a representative set of queries that includes natural-language questions, exact product or technical terms, identifiers, and short ambiguous queries. Adjust retrieval depth and the RRF constant only after inspecting failure cases, rather than tuning from a single query type.

  • Record which retrieval paths contributed to each final result.
  • Inspect documents promoted by only one path and documents supported by both.
  • Keep chunking and document identifiers consistent across dense and sparse indexes.
  • Treat fusion as a ranking layer that can be changed independently of indexing.