Why raw dense and sparse scores should not be added blindly

A dense-search score and a BM25 score are produced by different retrieval models and have different meanings. Their numeric ranges can shift with the corpus, query length, tokenization, embedding choice, and search configuration. A rule such as “add the two scores” can therefore make one retriever dominate for reasons unrelated to relevance.

Score normalization can be useful, but it requires measurement and ongoing monitoring. For a first hybrid retrieval implementation, rank-based fusion avoids assuming that scores from S3 Vectors dense retrieval and Quickwit BM25 are calibrated to one another. The application only needs each system to return an ordered list with stable document identifiers.

  • Run dense and BM25 retrieval for the same user query.
  • Request a bounded candidate list from each retriever.
  • Use a shared canonical document ID across both indexes.
  • Treat retrieval scores as system-specific signals, not universal relevance values.

Fuse the ranked lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its rank in each result list. For a document d, the fused score is the sum of 1 divided by k plus its rank, across the lists where it appears. The constant k reduces the difference between nearby ranks and prevents the first position in one list from overwhelming all other evidence.

For example, if a document ranks highly in both dense and BM25 results, it receives two contributions and tends to move upward in the fused order. A document appearing in only one list can still be returned, which preserves semantic-only and lexical-only matches. RRF does not require a document to have matching raw scores across retrievers.

  • Use one-based ranks: 1 for the first result, 2 for the second, and so on.
  • Compute: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Start with a fixed k and evaluate it on representative queries before changing it.
  • Deduplicate by canonical document ID before producing the final ranking.

Build an evaluation loop around real query classes

Hybrid retrieval should be evaluated against the query patterns your application actually receives. Separate exact-name, identifier, error-message, natural-language, and short ambiguous queries. BM25 may be especially important for literal strings, while dense retrieval may contribute more when users paraphrase source content or ask concept-level questions.

Keep the fusion layer observable. Log which retriever supplied each final candidate, its rank in each source list, and the final fused rank. Those records make it possible to diagnose failures: missing candidates indicate a retrieval-depth or indexing issue, while poor ordering among retrieved candidates may suggest tuning RRF, adjusting candidate counts, or adding a later reranking stage.

  • Create a small labeled query set before tuning fusion parameters.
  • Compare dense-only, BM25-only, and fused rankings on the same queries.
  • Inspect zero-result and low-quality-result queries separately.
  • Version the retrieval configuration so ranking changes can be traced and reviewed.