Why raw score blending is fragile

A dense-search score and a BM25 score do not necessarily share a common meaning or range. Their distributions can change with the embedding model, similarity metric, analyzer settings, document length, query wording, and corpus composition. Adding them with a fixed weight can therefore make a ranking sensitive to changes that are not obvious from the query layer.

This matters in an architecture such as Talqora's, where dense retrieval is backed by regional S3 Vectors and sparse retrieval uses Quickwit BM25. Each retrieval path can produce a useful ordered list, but its numeric scores should be interpreted in the context of that path. A rank-based method avoids assuming that a score of one value from one retriever is equivalent to the same value from another.

  • Dense retrieval is often helpful for paraphrases and conceptually related language.
  • BM25 is often helpful for exact names, codes, acronyms, and uncommon terms.
  • Raw-score weights can require repeated tuning as data and retrieval settings evolve.

Fuse lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each result list. For a document d, a common form is: RRF(d) = sum over retrievers of 1 / (k + rank_i(d)). The constant k reduces the difference between adjacent high ranks and prevents the first few positions from overwhelming every other signal.

To apply it, request a bounded candidate list from dense search and another from BM25, identify documents by a stable ID, then sum their RRF contributions. A document returned by both paths receives two contributions. A document returned by only one path can still rank well if it appears near the top of that path.

  • Choose a shared candidate depth, such as the top N results from each retriever.
  • Use a stable document or chunk ID when deduplicating results.
  • Keep k configurable so it can be evaluated against representative queries.
  • Sort by fused score, then use a deterministic tie-breaker such as document ID.

Make fusion observable before making it complex

Start by recording which retrieval path contributed to each fused result, along with its dense rank, BM25 rank, and final fused score. This makes it easier to inspect why a result appeared and to spot failure modes such as a sparse query producing no useful lexical matches or a semantic query being dominated by generic text.

Evaluate RRF on a small, curated query set that reflects production intent: exact lookup, troubleshooting language, multi-word concepts, and ambiguous terms. Compare the fused top results with each individual path. If a later system adds filters, reranking, or query routing, preserve this same per-stage visibility so ranking changes remain explainable.

  • Include queries containing identifiers and queries containing paraphrases.
  • Inspect overlap between dense and BM25 candidate lists.
  • Track cases where relevant items appear in only one retrieval path.
  • Apply metadata filters consistently before judging fused relevance.