Why raw dense and BM25 scores should not be added

A dense-search score reflects the relationship between query and document embeddings. A BM25 score is derived from term frequency, inverse document frequency, and document-length normalization. Even when both are useful relevance signals, the values do not inherently mean the same thing.

Adding the two scores directly creates an accidental weighting scheme. A change in embedding model, index configuration, corpus composition, or BM25 parameters can alter score distributions and silently shift which retrieval method dominates. This makes relevance behavior difficult to reason about and harder to debug.

  • Dense scores are model- and similarity-dependent.
  • BM25 scores depend on lexical statistics in the indexed corpus.
  • Score ranges can vary across queries as well as across systems.
  • Rank positions are usually more portable than raw score magnitudes.

Fuse ranked lists with reciprocal rank fusion

Reciprocal rank fusion (RRF) combines result lists using each document’s position rather than its original score. For every document returned by either search, assign a contribution of 1 divided by k plus its rank in each list. Sum those contributions, then sort documents by the resulting total.

The formula is: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the gap between adjacent top positions and prevents a single rank-one result from overwhelming all other evidence. A commonly used starting value is 60, but it is a relevance-tuning parameter rather than a universal rule.

  • Run dense retrieval against regional S3 Vectors.
  • Run lexical retrieval with Quickwit BM25 for the same query.
  • Normalize document identifiers before merging the lists.
  • Compute RRF scores, deduplicate documents, and return the highest-ranked results.

Make the fusion layer observable and easy to tune

Keep the two retrieval calls and the fusion step visible in application telemetry. For each returned document, record its dense rank, BM25 rank, fused rank, and whether it appeared in one or both lists. This makes it possible to distinguish a fusion issue from a candidate-generation issue.

Start with equal treatment for both lists and a fixed candidate depth, then evaluate using representative queries. Queries with product codes, quoted phrases, and uncommon terminology often reveal lexical strengths. Broad questions, paraphrases, and vocabulary mismatches can reveal dense-retrieval strengths. Tune list depths and k against judged results, not intuition alone.

  • Log ranks and source membership instead of relying only on final scores.
  • Use a stable document ID across dense and sparse indexes.
  • Test failure cases such as acronyms, typos, and near-duplicate content.
  • Apply filters consistently to both retrieval paths before fusion.