Why combine dense and sparse retrieval?

Dense retrieval represents text as vectors and is often valuable when a user’s wording differs from the wording in the indexed content. For example, a query about resetting account access may be related to a document that uses the phrase “credential recovery” rather than the same terms as the query.

BM25 is a sparse lexical ranking method. It can be especially useful when exact tokens carry meaning: product identifiers, error codes, names, acronyms, quoted phrases, and uncommon domain terminology. Neither signal should be assumed to cover every query type equally well.

  • Dense search contributes semantic neighborhood matching.
  • BM25 contributes exact-token and term-frequency signals.
  • Separate candidate lists preserve the strengths of each retriever.
  • Hybrid retrieval can be implemented as a ranking step after both searches.

Use reciprocal rank fusion instead of raw score addition

Dense similarity scores and BM25 scores are not inherently comparable. Their ranges, distributions, and meanings depend on the retrieval method and its configuration. Adding raw scores can therefore make one retrieval path dominate for numerical rather than relevance-related reasons.

Reciprocal rank fusion avoids direct score comparison. For each document in either result list, assign a contribution based on its rank: 1 divided by k plus the rank. Sum that contribution across lists, then sort documents by the combined value. The constant k reduces the difference between nearby top ranks and can be chosen as an application tuning parameter.

  • Retrieve a candidate set from regional S3 Vectors for the dense query.
  • Retrieve a candidate set from Quickwit BM25 for the lexical query.
  • For each document, sum 1 / (k + rank) for every list where it appears.
  • Deduplicate by a stable document or chunk identifier before returning the merged ranking.

Make fusion debuggable before tuning it

Store retrieval diagnostics with each query during evaluation: the dense rank, BM25 rank, fused rank, and identifiers for the returned chunks. These fields make it possible to see whether a result won because both retrievers agreed or because one retriever surfaced it alone.

Evaluate with a small set of representative queries before changing candidate counts or the RRF constant. Include queries with known identifiers, broad conceptual questions, short ambiguous terms, and organization-specific vocabulary. The goal is not to prove that one retriever always wins, but to understand where each ranking signal adds useful candidates.

  • Use stable IDs so the same chunk can be recognized across both result lists.
  • Log ranks rather than relying only on final fused order.
  • Inspect queries where dense and BM25 results have little overlap.
  • Keep retrieval and fusion configuration versioned alongside evaluation results.