Why raw dense and sparse scores should not be added

Dense retrieval scores describe proximity in an embedding space, while BM25 scores are produced from term-frequency and document-statistics signals. Even when both are useful rankings, their numeric ranges, distributions, and meanings differ.

Adding those scores directly creates a hidden calibration problem. A change in embedding model, corpus composition, BM25 configuration, or candidate depth can alter one score distribution and unintentionally change the balance of the hybrid result.

  • A high cosine-like similarity is not equivalent to a high BM25 score.
  • Per-query score normalization can still be unstable when candidate sets are small.
  • Rank positions are easier to compare than unrelated score scales.

Use RRF to merge ranked candidate lists

Run the dense query against regional S3 Vectors and the sparse query against Quickwit BM25 using the same logical retrieval filters. Keep a bounded, ordered candidate list from each path, then merge documents by their stable document identifier.

For every occurrence of a document at rank r, add 1 divided by k plus r to its fusion score. The constant k reduces the difference between adjacent top ranks; it is a tuning parameter rather than a universal relevance setting. Sort the merged documents by the resulting RRF score.

  • RRF(document) = Σ 1 / (k + rank_i(document)).
  • Use one-based ranks so the first result has rank 1.
  • Deduplicate by a shared document ID before returning results.
  • Choose a fixed candidate depth for each retriever and evaluate changes offline.

Make the fusion layer observable and easy to tune

Treat fusion as a small retrieval component with explicit inputs and outputs. Log the query, applied filters, dense rank, BM25 rank, fused rank, and whether a result appeared in one or both lists. This makes it possible to diagnose why a result moved without interpreting two unrelated raw scores.

Start with equal participation from both candidate lists. Then evaluate representative queries that include exact identifiers, uncommon terminology, paraphrases, and mixed-language or domain-specific phrasing relevant to your corpus. If one retriever is consistently absent for an important query class, investigate indexing, analysis, filters, and candidate depth before adding complex weighting.

  • Keep filters identical across dense and sparse searches where possible.
  • Record missing-side cases: dense-only, sparse-only, and overlapping candidates.
  • Version the embedding model, BM25 configuration, and fusion parameters together.
  • Use judged queries or downstream task outcomes to validate ranking changes.