Why raw score blending is fragile

A dense-search score and a BM25 score do not necessarily mean the same thing. Their ranges, distributions, and sensitivity to query length can differ. Even if each backend returns a numeric relevance value, adding the numbers together assumes a shared scale that may not exist.

That assumption creates brittle ranking behavior. A query containing a rare product code may produce a strong BM25 signal, while a broad natural-language question may benefit more from dense retrieval. A fixed weighted sum can overemphasize one source simply because its score range is larger.

  • Dense similarity depends on the embedding model and similarity metric.
  • BM25 depends on token statistics and term frequency behavior.
  • Score ranges can shift as documents, analyzers, or embedding models change.
  • A rank-based method avoids treating incomparable scores as interchangeable.

Fuse ranked lists with reciprocal rank fusion

With RRF, send the same user query to Talqora's dense retrieval path backed by regional S3 Vectors and to the sparse retrieval path backed by Quickwit BM25. Request a sufficiently deep candidate list from each path, then merge results by document identifier.

For every occurrence of a document at rank r in a list, add 1 divided by k plus r to its fusion score. The constant k reduces the difference between adjacent top ranks and keeps one list from dominating solely because of small rank changes. Sort the merged documents by their resulting RRF score.

  • Use a stable document or chunk ID to identify duplicates across result lists.
  • Start with the same candidate depth for both retrievers, such as the top N from each.
  • Compute: RRF(document) = sum of 1 / (k + rank) across lists containing the document.
  • Treat k and candidate depth as evaluation parameters, not universal constants.

Evaluate by query shape, not only an average

A fused ranking should be evaluated against representative queries. Include exact-name lookups, error messages, domain terms, conceptual questions, short queries, and longer queries. The useful question is not only whether fusion improves an aggregate metric, but whether it reduces the kinds of retrieval misses users notice.

Inspect disagreements between the two lists. If BM25 retrieves the right item because of an exact identifier, fusion should preserve it near the top. If dense retrieval finds a relevant paraphrase that BM25 misses, fusion should give that result a path into the final ranking. These examples also reveal whether chunking and metadata filters need attention before ranking changes.

  • Keep a small judged query set with relevance labels or reviewer notes.
  • Measure ranking quality at the result depth your application actually shows.
  • Log the source rank from dense retrieval and BM25 for fused results.
  • Re-evaluate after changes to chunking, embeddings, tokenization, or corpus composition.