Why ranks are safer to combine than raw scores

A dense-search similarity score and a BM25 score do not generally share a common scale. Their values can change with embedding choice, document length, tokenization, corpus composition, and the implementation of each retrieval system. Adding those scores directly assumes that a score of a given size means the same thing in both systems, which is rarely a safe assumption.

RRF avoids that assumption. It uses only each document's position in a ranked list. If a document appears near the top of both the dense and sparse results, it receives a stronger combined signal. If it appears in only one list, it can still be returned, but it receives less support from the fusion step.

  • Run dense and BM25 retrieval independently for the same query.
  • Request a candidate depth larger than the final number of results.
  • Use stable document or chunk IDs so results from both lists can be matched.
  • Keep retrieval scores available for debugging, but do not treat them as directly comparable.

Apply Reciprocal Rank Fusion in a small merge layer

For each result list, assign rank 1 to the first item, rank 2 to the second, and so on. Then calculate a fused score for every candidate: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across result lists in which document d appears. The constant k reduces the difference between adjacent top ranks and is commonly selected as a fixed tuning parameter.

For example, a chunk ranked third by dense retrieval and fifth by BM25 receives 1/(k+3) + 1/(k+5). A chunk found only at rank two in dense retrieval receives 1/(k+2). This gives consistent cross-list candidates an advantage without requiring score normalization.

  • Deduplicate by canonical chunk ID before producing the final ranking.
  • Assign no contribution for a list in which a candidate is absent.
  • Sort candidates by fused score descending, then use a deterministic tie-breaker.
  • Return the top N fused candidates to the next stage, such as context assembly or reranking.

Make fusion observable and tune it against real queries

Hybrid retrieval is easiest to improve when the merge layer records what happened. For a sampled query, log the dense rank, BM25 rank, fused rank, canonical ID, and selected content version for each returned candidate. This makes it possible to distinguish a retrieval miss from a fusion decision or a duplicate-ID problem.

Start with a representative evaluation set containing natural-language questions, exact identifiers, short queries, and domain-specific terminology. Compare dense-only, BM25-only, and RRF outputs using relevance judgments from your team. The goal is not to prove that one method always wins; it is to learn which query classes benefit from combining regional S3 Vectors dense search with Quickwit BM25.

  • Evaluate result quality separately for semantic questions and exact-match queries.
  • Inspect candidates that rank well in one list but disappear after fusion.
  • Version chunking and metadata rules alongside relevance evaluations.
  • Treat the RRF constant and candidate depths as configuration values, not permanent defaults.