Why raw-score mixing is fragile

Dense retrieval ranks documents by proximity between query and document vectors. Sparse BM25 retrieval ranks documents according to term-level evidence, including the query terms that appear in a document and their distribution in the collection. Both are useful signals, but their numeric outputs represent different scoring systems.

Adding or averaging those raw values assumes that a score from one retriever is comparable to a score from the other. That assumption can break as embedding models, corpus composition, tokenization, BM25 parameters, or query wording change. A document can be highly ranked by one system even when its raw score appears numerically smaller than a result from the other.

  • Dense search can recover semantic matches that use different wording.
  • BM25 can strongly favor exact names, identifiers, and uncommon terms.
  • Raw score ranges may vary across queries and retrieval systems.
  • Rank positions are easier to compare than unrelated score scales.

Fuse rankings with RRF

RRF starts with two or more independently ranked result lists. For each document, it adds a contribution based on the document's position in each list. The contribution is commonly written as 1 divided by k plus the document rank, where k is a positive constant chosen to reduce the influence of small rank differences near the top of a list.

For a document d, the fused score is: RRF(d) = sum over retrieval lists of 1 / (k + rank(d)). A document returned near the top by both dense search and BM25 receives a strong combined score. A document found by only one retriever can still appear, preserving useful semantic or exact-match results that the other retriever missed.

  • Retrieve a fixed candidate set from regional S3 Vectors for the dense path.
  • Retrieve a fixed candidate set from Quickwit BM25 for the sparse path.
  • Deduplicate candidates by a stable document or chunk identifier.
  • Sum each candidate's rank-based RRF contributions, then sort descending.

Make fusion observable and query-aware

Store enough retrieval metadata to explain why a result was selected: its dense rank, BM25 rank, fused rank, and which retrieval paths returned it. This makes it possible to inspect failures without treating the final ordering as a black box. It also helps distinguish a retrieval problem from a later generation or presentation problem.

Start with the same candidate depth for both retrievers, then review representative queries from your application. Queries containing product codes, quoted phrases, error messages, or proper nouns often need sparse evidence. Broad questions and paraphrases may benefit from dense evidence. RRF lets both routes contribute without requiring a universal conversion between their scores.

  • Log ranks and document identifiers for every fused result.
  • Test exact-term, paraphrase, multi-topic, and identifier-heavy queries separately.
  • Inspect documents surfaced by only dense search or only BM25.
  • Change candidate depth or fusion settings based on evaluated query sets, not a single example.