Why raw-score addition is risky

Dense retrieval and BM25 produce scores for different reasons. A dense score reflects a vector similarity calculation, while BM25 is driven by term frequency, document frequency, and length normalization. Even when both lists are useful, a score of 0.8 from one method does not inherently mean the same thing as 0.8 from the other.

Adding unlike scores can make one retriever dominate because of its numeric range rather than its relevance. The failure can be subtle: results may look plausible for common queries while exact identifiers, uncommon terminology, or wording-heavy requests are pushed down the final ranking.

  • Do not assume dense and BM25 scores have equivalent meaning.
  • Inspect score distributions across representative query types.
  • Treat a score-weighted blend as a calibration problem, not a default.

Fuse positions with reciprocal rank fusion

Rank-based fusion avoids comparing raw score scales. Retrieve a candidate list from dense search and another from Quickwit BM25, then assign each document a contribution based on its position in each list. A commonly used form is reciprocal rank fusion: RRF(d) = Σ 1 / (k + rank_i(d)).

In this formula, rank_i(d) is the one-based position of document d in a retrieval list, and k is a smoothing constant selected by the application. Documents appearing in both lists receive contributions from both, while a strong result from either retriever can still enter the merged ranking.

  • Request a bounded candidate set from each retriever.
  • Deduplicate documents using a stable document identifier.
  • Sum reciprocal-rank contributions for every returned document.
  • Sort by the fused score and return the top results.

Evaluate by query slice, not only by an aggregate

A fused ranking is only useful if it improves the queries your application actually receives. Build a small labeled set from real or representative requests, then include distinct slices such as exact product names, error codes, short natural-language questions, long descriptive questions, and acronym-heavy queries.

Compare dense-only, BM25-only, and fused lists using the same relevance labels. Aggregate metrics are useful, but slice-level review is where trade-offs become visible. For example, BM25 may be especially valuable when exact tokens matter, while dense retrieval may help when relevant content uses different wording.

  • Record the query, expected relevant documents, and query category.
  • Review top-ranked failures manually before changing fusion settings.
  • Keep candidate depths and fusion parameters versioned.
  • Re-run the evaluation set when embeddings, content, or analyzers change.