Why raw dense and BM25 scores should not be added

A dense retriever typically ranks documents using a vector similarity measure. A sparse BM25 retriever ranks documents from query-term and document-term statistics. Both produce numbers, but the numbers have different meanings, ranges, and distributions.

Adding those raw scores can make one retriever dominate for accidental reasons, such as a change in embedding normalization, a different similarity function, or the length distribution of indexed documents. Even normalizing scores per query can be fragile when a result list is short or contains a sharp score drop. A safer initial design is to fuse ranks rather than scores.

  • Dense retrieval is useful for semantic matches and paraphrases.
  • BM25 is useful for exact names, error codes, product terms, and rare tokens.
  • Score magnitude is not a shared measure of relevance.
  • Rank positions are directly comparable across result lists.

Apply Reciprocal Rank Fusion to two candidate lists

In Talqora, a hybrid request can obtain a dense candidate list from regional S3 Vectors and a sparse candidate list from Quickwit BM25. Keep each result's document identifier and its one-based rank. For every document appearing in either list, calculate an RRF score by summing 1 divided by k plus its rank for each list where it appears.

The constant k reduces the advantage of being ranked first over being ranked slightly lower. A commonly used starting value is 60, but it is a tuning parameter rather than a universal rule. After calculating fused scores, sort documents descending and return the top results. Documents retrieved by both methods receive contributions from both lists, while a strong result from only one method can still remain competitive.

  • Retrieve the top N candidates from dense search and the top N from BM25.
  • For document d, use RRF(d) = Σ 1 / (k + rank_i(d)).
  • Assign no contribution from a retriever that did not return the document.
  • Deduplicate by a stable document or chunk identifier before final sorting.

Choose candidate depth and evaluate the fused result

Candidate depth is a recall decision. If each retriever returns too few results, fusion cannot recover relevant documents that were never included. Start with a depth that comfortably exceeds the number of results shown to a user, then adjust it using representative queries. The right value depends on corpus size, chunking strategy, query types, and downstream reranking or generation steps.

Evaluate dense-only, BM25-only, and fused retrieval on the same labeled query set. Include semantic questions, exact-phrase lookups, identifiers, misspellings, and mixed queries. Review failures by query class rather than relying only on an aggregate metric: fusion is most valuable when each retriever covers relevant items the other misses.

  • Log per-retriever ranks alongside the final fused rank for debugging.
  • Test whether relevant documents appear in either candidate list before tuning fusion.
  • Keep query preprocessing consistent with the expectations of each retriever.
  • Consider a later reranking stage only after candidate recall is reliable.