Why raw hybrid scores are difficult to combine

Dense retrieval and BM25 produce scores with different meanings. A dense-search score is derived from the relationship between query and document embeddings, while BM25 reflects term occurrence and document-level statistics. Even when both lists are useful, their numeric ranges are not inherently comparable.

A weighted sum can work when scores have been carefully calibrated for a known corpus and query distribution. In practice, corpus changes, embedding-model changes, analyzer configuration, and query mix can all alter score distributions. Treating a score of 0.8 from one system as equivalent to a score of 0.8 from another is usually an assumption that needs evidence.

  • Dense search can recover conceptually related content with different wording.
  • BM25 can strongly reward exact identifiers, error messages, product names, and rare terms.
  • Raw-score fusion adds a normalization problem before it solves a ranking problem.

Fuse ranks instead of scores with RRF

RRF assigns each document a contribution based on its position in each ranked list. For a document d, a common formulation is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is its one-based position in list i and k is a positive constant. Documents appearing near the top of either list receive more credit, while documents supported by both lists accumulate credit.

The important property is that RRF only requires ordered results. An application can request a candidate list from Talqora's dense retrieval path and a candidate list from its Quickwit BM25 path, join results by a shared document or chunk identifier, calculate fusion scores, and sort the merged candidates.

  • Use stable, shared IDs for the same chunk in dense and sparse indexes.
  • Keep ranks one-based when implementing the formula.
  • Choose a fixed k initially, then evaluate it against representative queries.
  • Fetch enough candidates from each retriever to give fusion meaningful overlap and alternatives.

Make fusion observable before tuning it

Start with a small evaluation set that reflects actual search behavior: exact lookup queries, natural-language questions, short ambiguous queries, and terminology-heavy requests. For each query, record the top results from dense retrieval, BM25, and the fused ranking. This makes it possible to see whether fusion is recovering useful documents or merely reshuffling already strong results.

Operationally, log the source ranks that contributed to each fused result. A result that ranks well because it was first in BM25 tells a different story from one that was moderately high in both lists. Those signals help diagnose missing metadata, inconsistent chunking, vocabulary gaps, or embedding coverage issues without treating the fusion formula as a black box.

  • Compare dense-only, BM25-only, and fused top-k relevance on the same query set.
  • Inspect queries where the two lists have little or no overlap.
  • Track document IDs and chunk versions so joins remain correct after reindexing.
  • Tune chunking and indexing quality before relying on fusion weights to compensate for weak candidates.