Why raw dense and BM25 scores should not be added casually

A dense-search score and a BM25 score are produced by different ranking models and often have different ranges, distributions, and meanings. A larger value in one system does not necessarily represent the same amount of relevance as a larger value in the other. Adding those values directly can make the final ranking depend more on score scale than on evidence of relevance.

Score normalization can be useful, but it requires careful validation across query types, document lengths, and index changes. RRF avoids this first problem by consuming only each result’s position in a ranked list. A document that appears near the top of either list earns credit, and a document appearing near the top of both lists earns more.

  • Run dense and BM25 retrieval independently for the same query.
  • Retain document IDs and one-based rank positions from each result list.
  • Do not treat score values from separate retrieval systems as inherently comparable.
  • Use a stable document identifier to join results across the two lists.

Apply reciprocal rank fusion at the application layer

For each candidate document d, calculate an RRF score by summing 1 / (k + rank) for every ranking in which d appears. The constant k reduces the difference between adjacent top positions and prevents a single rank-one placement from overwhelming the combined result. A commonly used starting point is k = 60, though it should be treated as a tunable parameter rather than a universal rule.

For example, if a document ranks 2nd in dense retrieval and 5th in BM25, its fused score is 1 / (k + 2) + 1 / (k + 5). A document found only by one retriever can still appear in the final results, while documents supported by both retrievers naturally move upward. This makes RRF well suited to a system that queries S3 Vectors for dense candidates and Quickwit for lexical candidates.

  • Retrieve a bounded candidate set from each source, such as the top N results.
  • Assign rank 1 to the first result in each list.
  • Sum the RRF contribution for each document ID across lists.
  • Sort candidates by fused score and return the desired final top K.

Tune candidate depth and validate with representative queries

RRF can only promote documents that enter at least one candidate list. Candidate depth therefore matters: if dense and BM25 retrieval each return too few results, relevant documents may never reach fusion. Start with a candidate depth larger than the final result count, then evaluate whether deeper retrieval changes the quality of the final list for your workload.

Validation should include queries with exact identifiers, uncommon product names, natural-language questions, and ambiguous phrasing. Exact-match queries often reveal the value of BM25, while paraphrased questions can reveal the value of dense retrieval. Review not only aggregate relevance judgments but also failure cases, because they show whether a retriever is missing candidates or whether fusion is ordering good candidates poorly.

  • Keep the final response size separate from per-retriever candidate depth.
  • Log source ranks for returned documents to make ranking decisions inspectable.
  • Test k and candidate depth together using a fixed evaluation set.
  • Re-evaluate after changing embeddings, analyzers, document chunking, or corpus composition.