Why adding dense and BM25 scores is risky

A vector similarity score and a BM25 relevance score are produced by different ranking systems. Their ranges, distributions, and sensitivity to query length can differ substantially. A score of 0.8 from one retrieval method does not inherently mean the same thing as 0.8 from another.

Directly adding or weighting raw scores can still work after careful calibration, but it creates operational overhead. Changes to embeddings, document chunking, BM25 configuration, or corpus composition can change score behavior and require the weighting logic to be revisited.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact language, product names, codes, and rare terms.
  • Raw scores are ranking signals, not a shared measurement scale.
  • A fusion method based on rank avoids assuming score equivalence.

Fuse ranked lists with reciprocal rank fusion

Reciprocal rank fusion (RRF) combines result lists using each document's position rather than its original score. For every document returned by either retrieval path, add a contribution of 1 divided by k plus its rank in that list. Sort documents by the resulting total.

The constant k reduces the difference between adjacent high ranks and makes the fusion less dominated by a single first-place result. A commonly used starting value is 60, but it is a tuning choice rather than a universal rule. Keep the same k while evaluating changes so comparisons are meaningful.

  • Run dense retrieval against regional S3 Vectors.
  • Run sparse retrieval with Quickwit BM25 using the same user query or a query-specific lexical form.
  • Assign ranks starting at 1 within each returned list.
  • Compute RRF(document) = sum of 1 / (k + rank) across lists where the document appears.

Implement fusion after candidate retrieval

Request a bounded candidate set from each retriever, then deduplicate by a stable document or chunk identifier before calculating the fused score. A document present in both lists receives two contributions, which is often desirable: it has evidence from both semantic and lexical matching.

Evaluate fusion using representative queries rather than only aggregate relevance labels. Include exact identifiers, vague natural-language questions, short queries, multilingual queries where applicable, and queries whose wording differs from the source text. Inspect not only whether a relevant item appears, but whether it appears early enough for the downstream experience.

  • Use a stable ID so the same chunk from both systems is merged correctly.
  • Record each source rank alongside the fused rank for debugging.
  • Set candidate-list depths high enough to create useful overlap, then control final result count separately.
  • Treat k, candidate depth, chunking, and query construction as evaluation variables.