Why raw dense and BM25 scores should not be added directly

A dense retriever and a BM25 retriever produce scores with different meanings. A dense score may reflect a vector similarity measure, while BM25 is a term-statistics ranking function influenced by document frequency, length normalization, and query terms. Even when both scores are numerically bounded, equal-looking values do not necessarily represent equal relevance.

Adding uncalibrated scores can make ranking unstable. A small change in embedding model, corpus composition, BM25 parameters, or retrieval depth may alter score distributions and unexpectedly shift which retrieval method dominates. This is especially risky when queries range from natural-language questions to exact strings such as ticket IDs or stack-trace fragments.

  • Dense retrieval helps with paraphrases, concepts, and related terminology.
  • BM25 helps with exact terms, rare tokens, codes, and literal phrases.
  • Score scales are retrieval-system-specific, not a shared relevance unit.
  • A merge strategy should preserve each retriever’s ordering without assuming score comparability.

Fuse ranked lists with Reciprocal Rank Fusion

Reciprocal Rank Fusion, commonly abbreviated as RRF, combines ranked lists rather than their original scores. For each document, add a contribution from every list in which it appears: 1 divided by k plus the document’s rank. The constant k reduces the impact of small rank differences near the top of a list.

For a Talqora-backed retrieval flow, request a candidate list from dense search over S3 Vectors and another from Quickwit BM25. Deduplicate by a stable document or chunk identifier, calculate the RRF total, and sort descending. The resulting list becomes the candidate set for a user-facing search experience or a later reranking stage.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first result has rank 1.
  • Start with the same candidate depth for dense and sparse retrieval.
  • Choose a stable chunk ID so the same content can be merged across both result lists.

Make fusion observable and tune it with real queries

RRF is simple, but it still needs operational discipline. Log the dense rank, sparse rank, fused rank, and retrieval source for each returned item. These fields make it possible to diagnose whether a result was supported by both methods, rescued by one method, or promoted because candidate sets contain overlapping chunks.

Evaluate with a query set that reflects production traffic. Include broad conceptual questions, short keyword searches, exact identifiers, quoted phrases, and domain-specific terminology. Review not only top-one relevance but also whether the final top-k contains the supporting passages needed for the next step in the application.

  • Keep retrieval depth separate from the number of results shown to a user.
  • Inspect queries where dense-only and BM25-only results disagree.
  • Track document IDs and source ranks in logs for reproducible debugging.
  • If later adding a reranker, apply it to the fused candidate set rather than relying on either retriever alone.