Why raw dense and BM25 scores should not be added directly

A dense retrieval score represents a relationship in embedding space, while a BM25 score is derived from term frequency, document frequency, and document length. Even when both are useful signals, their numeric ranges and distributions are not inherently comparable.

Adding those scores directly creates a hidden tuning problem. A change in embedding model, query vocabulary, corpus composition, or BM25 configuration can shift one score distribution and quietly change which retrieval method dominates the combined ranking.

  • Dense search is often valuable when query and document language differ but mean similar things.
  • BM25 is often valuable for exact terminology, product names, codes, and uncommon tokens.
  • A stable hybrid strategy should combine evidence without treating unrelated score scales as equivalent.

Fuse ranks, not scores, with RRF

RRF assigns each document a contribution based on its position in each result list. For a document d, the fused score can be written as: RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs across retrieval lists in which the document appears, and rank_i(d) starts at 1.

The constant k reduces the difference between adjacent positions near the top of a list. Rather than requiring a universal score threshold, RRF asks a simpler question: did this document rank well in dense retrieval, sparse retrieval, or both?

  • Run dense search against Talqora's regional S3 Vectors-backed retrieval path.
  • Run sparse search with the Quickwit BM25 path using the same query or a query variant.
  • Keep a document identifier with every hit, merge duplicate identifiers, and sum their RRF contributions.
  • Sort descending by fused score and return the top N documents to the next stage.

Choose retrieval depth deliberately and inspect disagreements

RRF can only promote documents that appear in at least one candidate list. Request enough candidates from each retriever to give the fusion stage useful coverage, especially when the final application needs only a small number of results. The best depth depends on corpus size, query behavior, and downstream context limits, so it should be evaluated on representative queries rather than chosen once by intuition.

Disagreement between dense and sparse results is useful diagnostic information. If BM25 repeatedly surfaces the correct document while dense retrieval misses it, inspect tokenization, chunk boundaries, metadata, and embedding coverage. If dense retrieval finds relevant paraphrases that BM25 misses, preserve that semantic path rather than over-tuning keyword queries.

  • Log each candidate's dense rank, BM25 rank, and fused rank for sampled queries.
  • Evaluate known-answer queries containing exact identifiers as well as natural-language paraphrases.
  • Use the same canonical document or chunk ID across both indexes so fusion can reliably deduplicate results.
  • Treat RRF as a transparent first-stage combiner; later reranking can be added only when its value is validated.