Why raw dense and sparse scores should not be added directly

A dense retriever typically ranks documents by a vector similarity measure, while BM25 ranks documents using term frequency, document frequency, and document-length normalization. Even when both systems return a numeric score, those values are not automatically on a shared scale.

Adding raw scores can make a fusion policy sensitive to implementation details, index settings, query length, or future changes in either retrieval layer. A rank-based approach avoids treating a similarity score and a BM25 score as equivalent quantities.

  • Dense retrieval can surface semantically related phrasing.
  • BM25 can preserve exact terms, identifiers, product names, and rare tokens.
  • Raw score ranges may differ across retrievers and query types.
  • Rank positions are a simpler common interface for fusion.

Fuse two candidate lists with RRF

Run dense retrieval against the vector index and BM25 retrieval against the sparse index, requesting a candidate list from each. For every document that appears in either list, calculate an RRF score by summing 1 divided by k plus that document’s rank in each list.

The formula is RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the difference between adjacent top ranks, preventing one result list from dominating solely because a document appears at rank 1 rather than rank 3. Keep k configurable and evaluate it with representative queries rather than assuming one value is universally best.

  • Use one-based ranks: rank 1 is the first result in a list.
  • Assign no contribution when a document is absent from a retriever’s candidate list.
  • Deduplicate by a stable document identifier before sorting.
  • Sort the union of candidates by descending RRF score.

Make fusion debuggable and evaluate it by query class

Log the dense rank, BM25 rank, and final fused rank for returned documents. This makes it possible to explain whether a result won because both retrievers agreed, because BM25 found an exact match, or because dense retrieval recovered relevant terminology expressed differently.

Evaluate hybrid behavior with query groups that reflect the corpus: natural-language questions, acronym-heavy requests, exact identifier lookups, and queries containing uncommon terms. Review not only aggregate relevance judgments but also cases where one retriever consistently contributes useful unique results.

  • Store per-retriever rank contributions alongside the fused score.
  • Inspect queries where dense and sparse top results have little overlap.
  • Set candidate depths high enough to give fusion meaningful alternatives.
  • Re-test fusion after changing chunking, embedding models, or BM25 indexing settings.