Why raw dense and BM25 scores should not be added blindly

Dense retrieval typically ranks documents by the similarity between query and document vectors. BM25 ranks documents using lexical term statistics, including term frequency and document frequency. Even when both systems return numeric scores, those values are not inherently interchangeable.

Adding the two scores directly can make one retriever dominate merely because its numeric range is larger or more variable. The issue can also change across query types: a short identifier-heavy query and a natural-language question may produce very different score distributions.

Before combining raw scores, a team would need a well-defined normalization and validation strategy. For many applications, rank-based fusion is a safer initial design because it uses ordering rather than assuming score equivalence.

  • Dense similarity and BM25 scores represent different ranking signals.
  • Score ranges may vary by query, corpus, and retrieval implementation.
  • Direct score addition requires explicit normalization and evaluation.
  • Rank fusion can combine candidate lists without calibrating score scales.

Fuse ranked lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank for every list in which it appears: RRF(d) = Σ 1 / (k + rank_i(d)). Documents found by both dense and sparse retrieval receive contributions from both lists.

The constant k dampens the difference between adjacent rank positions and prevents the first few positions from overwhelming the rest of the candidate set. It is a tunable parameter, not a universal truth; keep it explicit in configuration and evaluate it with representative queries.

Use stable document identifiers when merging results. A document returned by both S3 Vectors-backed dense retrieval and Quickwit BM25 retrieval must resolve to the same identifier before its rank contributions can be accumulated.

  • Retrieve the top N candidates from dense search and top N from BM25.
  • Map both result sets to a shared canonical document ID.
  • Add a reciprocal-rank contribution for every occurrence.
  • Sort descending by the accumulated RRF score and return the desired top K.

Make hybrid retrieval observable and testable

Log the source ranks that produced each final result, not just its fused position. For example, record whether a result came from dense retrieval only, BM25 only, or both, along with its rank in each source list. This makes it easier to diagnose surprising results and identify query classes that favor one retriever.

Build a small evaluation set from real tasks: questions requiring semantic matching, exact names or codes, recent terminology, and documents with important phrases. Compare dense-only, sparse-only, and RRF-fused rankings using the same candidate depth and relevance judgments.

RRF is a candidate-fusion method, not a replacement for downstream ranking decisions. If an application needs metadata constraints, apply those consistently during retrieval or filtering. If it later adds a reranking stage, preserve the fused candidate provenance so the full retrieval path remains inspectable.

  • Record dense rank, BM25 rank, and fused rank for returned documents.
  • Evaluate candidate depth separately from the final number of results shown.
  • Include exact-match and semantic-match queries in relevance tests.
  • Treat fusion parameters as versioned retrieval configuration.