Why raw dense and BM25 scores should not be added

A dense-search score represents the relationship between embedding vectors under the selected similarity measure. A BM25 score is produced from term-frequency and corpus-statistics signals. Even when both are useful ranking signals, the numbers do not inherently express the same quantity or range.

Adding raw scores assumes that a one-point change in one system has meaning comparable to a one-point change in the other. That assumption can quietly over-weight one retriever, especially as indexes, analyzers, embedding models, or query composition change. A rank-based merge avoids making that assumption.

  • Keep dense and sparse retrieval as separate candidate-generation steps.
  • Request a bounded ranked list from each retriever for the same query.
  • Use a stable document or chunk identifier to recognize duplicates across lists.
  • Treat retriever scores as local ordering signals unless they have been explicitly calibrated.

Apply reciprocal rank fusion at the application layer

RRF assigns each document a contribution based on its position in each result list. For a document d, a common form is RRF(d) = sum over lists of 1 / (k + rank(d)). The constant k reduces the impact of very high positions and makes the merge less sensitive to small rank changes.

For each user query, retrieve dense candidates from regional S3 Vectors and sparse candidates from Quickwit BM25. Build a map keyed by document or chunk ID, add the RRF contribution for every list in which an item appears, then sort the map by the fused score. A result present in both lists receives two contributions; a result unique to one list can still appear if it ranks well there.

  • Use one-based ranks: the first result has rank 1.
  • Choose k as a configurable application parameter rather than a universal constant.
  • Deduplicate before presenting results, while retaining the strongest available metadata for the item.
  • Keep the original dense and BM25 ranks in logs for diagnosis.

Evaluate fusion with query classes, not one aggregate impression

Hybrid retrieval is most useful when evaluated against the kinds of queries the application actually receives. Create a small labeled set containing exact-name or ID queries, terminology-heavy queries, paraphrased natural-language questions, and ambiguous short queries. Compare dense-only, BM25-only, and fused rankings against the same relevance judgments.

When a fused result looks wrong, inspect its two component ranks. This distinguishes a dense-retrieval issue from a sparse-retrieval issue and from a fusion-policy issue. For example, an exact identifier may justify stronger sparse coverage, while a paraphrase failure may point to embedding, chunking, or query construction rather than RRF itself.

  • Measure whether a relevant item appears in the candidate set before judging final ordering.
  • Review failures by query type and by document or chunk length.
  • Version the fusion setting alongside embedding, indexing, and analyzer changes.
  • Use evaluation findings to adjust candidate-list depth or fusion policy deliberately.