Why raw dense and sparse scores should not be added

A dense-search score reflects the relationship between a query embedding and document embedding. A BM25 score reflects term statistics such as query-term frequency, document-term frequency, and corpus-wide rarity. Even when both systems return larger values for more relevant documents, the numerical ranges and distributions do not share a common meaning.

Adding scores directly can make one retrieval path dominate for accidental reasons. A change to embedding generation, document chunking, BM25 parameters, or corpus composition may alter score distributions without changing the underlying relevance intent. This creates a ranking rule that is difficult to reason about and fragile to maintain.

  • Dense retrieval is useful for semantic similarity and paraphrased queries.
  • BM25 is useful for literal terms, product names, codes, and rare vocabulary.
  • A hybrid layer should combine evidence without assuming both score scales are equivalent.

Fuse ranked lists with Reciprocal Rank Fusion

RRF starts with two ranked candidate lists: one from dense search and one from BM25. For each document, assign a contribution based on its position in each list, then sum those contributions. A common form is RRF(d) = Σ 1 / (k + rankᵢ(d)), where k is a positive constant and rankᵢ(d) is the document's one-based position in list i.

The method rewards documents that appear near the top of either list and gives additional credit to documents supported by both. Because it uses position instead of raw relevance values, it avoids the need to normalize dense and sparse scores into a shared scale before combining them.

  • Retrieve a bounded candidate set from regional S3 Vectors for dense search.
  • Retrieve a bounded candidate set from Quickwit BM25 for sparse search.
  • Deduplicate documents by a stable document or chunk identifier.
  • Compute the RRF sum for each candidate, then sort by the fused score.

Make fusion behavior observable and testable

RRF is simple, but its surrounding retrieval pipeline still needs deliberate choices. Decide whether dense and sparse search should return the same number of candidates, ensure both paths address the same document population where appropriate, and define how deleted or unavailable records are filtered after fusion. Stable identifiers are essential because the same content may be returned by both systems under different internal result formats.

Evaluate with representative queries before treating a fusion setting as a default. Include exact-match queries, acronym-heavy queries, natural-language questions, and queries where relevant content uses different wording. Inspect not only the top result but also which retrieval path contributed each candidate; that provenance makes ranking regressions easier to diagnose.

  • Log dense rank, BM25 rank, and fused rank for returned candidates.
  • Keep candidate-list sizes and the RRF constant as explicit configuration.
  • Test empty results from either retrieval path.
  • Use relevance judgments or review sets to compare ranking changes over time.