Why one retrieval method is rarely enough

Dense retrieval represents queries and documents as vectors, making it well suited to semantic matches. A query such as “how do I rotate access credentials?” can still retrieve material titled “credential renewal procedure,” even though the wording is not identical.

Sparse BM25 retrieval scores the terms present in the query and documents. It is often valuable for queries containing distinctive text: error codes, model numbers, internal abbreviations, API field names, or exact names. In these cases, semantic similarity alone may not preserve the importance of the literal token.

Running both retrieval paths gives the ranking layer two independent signals. The remaining challenge is combining those signals without assuming that a dense score and a BM25 score are directly comparable.

  • Use dense retrieval for meaning, paraphrases, and concept-level recall.
  • Use BM25 for lexical precision and rare query terms.
  • Treat raw scores from different retrieval systems as separate scales.

Fuse ranked lists instead of normalizing raw scores

Reciprocal rank fusion, commonly abbreviated as RRF, combines result lists by position rather than by raw score. For each document, add 1 divided by k plus its rank for every list in which it appears. A document that ranks well in both dense and BM25 results receives a stronger combined score.

The formula is RRF(d) = Σ 1 / (k + rank_i(d)). The constant k dampens the difference between adjacent ranks and prevents the first few positions in a single list from overwhelming every other signal. The important operational detail is that ranks begin at 1, not 0.

Because RRF uses ordinal positions, it avoids fragile score calibration between S3 Vectors dense results and Quickwit BM25 results. This makes it a useful first fusion strategy when the two backends produce scores with different meanings or ranges.

  • Request a candidate list from dense search and another from BM25.
  • Assign rank 1 to the first item in each list.
  • Add reciprocal-rank contributions for documents appearing in one or both lists.
  • Sort by the fused score and return the top results.

Implement fusion with stable IDs and inspect the misses

A fusion layer depends on a stable document or chunk identifier shared by both indexes. If dense retrieval returns a chunk ID that BM25 cannot also reference, the application cannot recognize that the two results represent the same retrieval unit. Decide whether the unit of retrieval is a whole document, a passage, or another chunking boundary, then keep that choice consistent.

Start with a modest candidate depth from each retriever and record the source ranks alongside the fused rank. These fields make debugging concrete: a surprising result may have come from high dense rank, high BM25 rank, or agreement between both. They also reveal when one retrieval path is returning irrelevant candidates.

RRF is a ranking step, not a substitute for corpus preparation. Clean text, coherent chunks, useful metadata, and a clear filtering policy still determine which documents are eligible to be retrieved. Apply required filters consistently before fusion so the fused list does not mix results from incompatible scopes.

  • Use the same canonical ID in dense and sparse indexes.
  • Store dense rank, BM25 rank, and fused score for evaluation and debugging.
  • Apply tenant, access, language, or document-status filters consistently.
  • Evaluate queries with exact terms and paraphrased intent, not only one query style.