Why raw dense and BM25 scores should not be added

A dense-search score represents the relationship between query and document embeddings under a chosen similarity function. A BM25 score is derived from term frequency, document frequency, field length, and query terms. Even when both systems return higher scores for stronger matches, their numeric ranges and distributions have different meanings.

Adding or averaging those values can make one retrieval method dominate for reasons unrelated to relevance. A change to the embedding model, vector similarity configuration, corpus composition, BM25 analyzer, or indexed field can shift score distributions and silently change hybrid behavior.

  • Treat dense and sparse scores as system-specific signals, not a shared relevance scale.
  • Do not assume a score of 0.8 from one method is comparable to 0.8 from another.
  • Evaluate score normalization separately if your application requires it.
  • Prefer a method that depends on result order when starting with hybrid retrieval.

Fuse ranked lists with Reciprocal Rank Fusion

RRF assigns a document a contribution based on its position in each ranked result list. For every list in which a document appears, add 1 divided by k plus the document rank. The constant k reduces the difference between nearby top ranks and prevents a single rank-one result from overwhelming all other evidence.

For a query, retrieve a candidate list from regional S3 Vectors for dense search and a candidate list from Quickwit BM25 for sparse search. Deduplicate documents by a stable document ID, calculate each document's RRF score, sort descending, and return the top fused results.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first item in a list has rank 1.
  • A commonly used starting value for k is 60, then validate it against representative queries.
  • Retrieve more candidates than the final page size so both methods can contribute before fusion.

Make fusion observable and tune it with real queries

RRF is intentionally simple, but it still needs evaluation. Build a small query set that includes natural-language questions, exact identifiers, abbreviations, misspellings where relevant, and multi-concept requests. For each query, inspect dense-only, BM25-only, and fused results side by side.

Log the source ranks that produced every fused result. This makes it possible to see whether a document was supported by both retrieval methods, rescued by sparse matching for an exact term, or introduced by semantic similarity. Those observations are more actionable than looking at the fused score alone.

  • Track whether relevant documents appear in dense, sparse, or both candidate lists.
  • Inspect failures caused by missing candidates before changing fusion settings.
  • Keep document IDs and filtering rules consistent across both retrieval paths.
  • Use offline relevance judgments and production feedback to decide candidate depths and k.