Why raw dense and sparse scores should not be added blindly

A dense retriever typically returns a vector-similarity score, while BM25 returns a term-matching relevance score. Even when both lists are sorted correctly on their own, their numeric ranges and distributions are not inherently comparable. A score of 0.8 from one system does not automatically mean the same thing as a score of 8 from another.

Normalizing scores can be useful, but it introduces choices about ranges, query-dependent distributions, outliers, and weighting. Those choices can become fragile as embeddings, analyzers, document collections, or query patterns change. Rank-based fusion avoids treating the two score scales as if they were a shared measurement.

  • Dense search is useful for semantic and paraphrase-style matches.
  • BM25 is useful when exact terms, identifiers, and rare tokens matter.
  • Each retriever should return a ranked candidate list before fusion.
  • Keep the source rank and source score for debugging, even if fusion uses rank.

Fuse candidate lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in each result list. For a document d, compute RRF(d) as the sum of 1 divided by k plus rank(d) across the lists where d appears. The constant k reduces the influence of small rank differences near the top of a list.

For example, request the top N dense candidates from regional S3 Vectors and the top N sparse candidates from Quickwit BM25. Deduplicate document IDs, add the RRF contribution from each list, sort by the fused value, and return the top results. A document that ranks well in both lists rises naturally, while a strong result from only one retriever can still remain competitive.

  • Use one-based ranks: the first result has rank 1.
  • Start with the same candidate depth for dense and sparse retrieval.
  • Choose a fixed k and evaluate it on representative queries before changing it.
  • Deduplicate on a stable document or chunk identifier, not on display text.

Make fusion observable before tuning it

Hybrid retrieval is easier to improve when the fused result explains its origin. For every returned candidate, record whether it came from dense retrieval, sparse retrieval, or both; retain its rank in each list; and store the final fused rank. This makes it possible to inspect whether a poor result was caused by candidate generation, fusion, or later reranking.

Evaluate with queries that represent real retrieval pressure: acronym-heavy searches, error messages, product names, natural-language questions, and queries with mixed terminology. If exact identifiers routinely disappear, inspect sparse candidate depth and field analysis. If paraphrases are missed, inspect embedding inputs and dense candidate depth. Change one variable at a time so that improvements remain attributable.

  • Log query, candidate ID, dense rank, sparse rank, and fused rank.
  • Review queries where the desired document appears in only one candidate list.
  • Measure retrieval quality before evaluating any downstream generation step.
  • Treat candidate depth as a recall control and final result count as a presentation control.