Why raw dense and BM25 scores are difficult to add

Dense retrieval returns documents based on vector similarity, while BM25 produces a lexical relevance score derived from term statistics. Even when both methods retrieve the same document, their numeric scores do not inherently mean the same thing. A score of 0.8 from one retriever is not automatically comparable to a score of 0.8 from another.

Score distributions can also change with embedding choice, document length, query wording, corpus composition, and retrieval configuration. Adding uncalibrated scores can quietly bias results toward whichever retriever happens to emit larger numbers, rather than toward the retriever with the more useful ranking for the query.

  • Dense search can recover semantically related wording.
  • BM25 can strongly reward exact terms, identifiers, and uncommon phrases.
  • Raw scores should not be treated as interchangeable without calibration.
  • Rank positions are easier to compare across independent retrievers.

Fuse ranked lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in each ranked list. For a document d, sum 1 divided by k plus its rank across the available retrievers: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the gap between adjacent top ranks and prevents a single first-place result from overwhelming all other evidence.

In a Talqora retrieval flow, request a candidate list from dense search and a candidate list from Quickwit BM25, identify documents by a stable shared ID, then aggregate their RRF contributions. A document returned by both methods receives two contributions; a document found by only one method can still rank well if it appears near that list's top.

  • Use the same stable document or chunk ID in both result lists.
  • Choose a candidate depth large enough to allow overlap and discovery.
  • Apply the same k value consistently while evaluating a configuration.
  • Sort documents by descending fused score after aggregation.

Evaluate fusion with query classes, not one average

RRF is intentionally simple, but it still needs evaluation against representative queries. Build a small labeled set that includes exact product names, error messages, abbreviations, conceptual questions, and queries that mix an identifier with natural-language intent. These classes reveal when BM25, dense retrieval, or their combination is carrying the result.

Inspect more than a single aggregate metric. Compare whether a known relevant item appears in the top few results, whether hybrid retrieval introduces duplicate chunks, and whether exact-match queries remain precise. If needed, change candidate depths or k and repeat the same evaluation. Keep the fusion rule understandable so ranking changes can be traced back to an engineering decision.

  • Track top-k relevance separately for lexical and semantic query classes.
  • Review queries where dense-only and BM25-only results disagree.
  • Deduplicate by canonical document or chunk identifier before presentation.
  • Retain retrieval provenance to see which list contributed to a result.