Why raw-score blending is fragile

A dense retriever ranks documents by vector similarity or distance. BM25 ranks them from term frequencies, document frequencies, and document-length normalization. Even when both outputs are represented as numbers where larger means better, a score of 0.8 from one retriever does not have the same meaning as 0.8 from the other.

Score distributions can also change with the query. A highly specific keyword query may produce a strong BM25 leader, while a paraphrased or vocabulary-mismatched query may be better served by dense retrieval. A fixed weighted sum of raw scores can therefore make relevance depend more on score scale than on the evidence each retriever found.

  • Dense search can retrieve semantically related text that lacks the query's exact terms.
  • BM25 can strongly reward exact identifiers, rare terminology, and phrase overlap.
  • Raw scores may vary by index, query, analyzer, embedding model, and similarity metric.
  • Rank order is often more stable and easier to combine than raw score magnitude.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its rank in each result list, then adds those contributions together. For a document d, the fused score is: RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs across retrievers that returned d, rank_i is the document's one-based rank, and k is a positive constant that reduces the influence of small rank differences.

In a Talqora retrieval flow, query regional S3 Vectors for a dense candidate list and Quickwit BM25 for a sparse candidate list. Identify matching documents using a stable shared document ID, calculate the RRF total for the union of candidates, sort descending, and return the top fused results. A document appearing in both lists is rewarded, while a document highly ranked by only one retriever can still be surfaced.

  • Request a bounded candidate set from each retriever, such as the top N results.
  • Use one canonical document ID across dense and sparse indexing paths.
  • Choose a k value deliberately and keep it observable in configuration.
  • Preserve source ranks and the fused score in debug output for inspection.

Evaluate fusion as a retrieval change

RRF is not a substitute for evaluation. Build a query set that reflects production traffic and includes exact-name lookups, acronym-heavy queries, natural-language questions, and cases where vocabulary differs between the query and the relevant text. For each query, record whether the expected document appears in the returned set and where it ranks.

Compare dense-only, BM25-only, and fused retrieval using the same corpus, filters, and result depth. Review failures manually as well as through aggregate metrics. If exact-match queries lose important results, inspect sparse indexing and candidate depth. If paraphrase queries fail, inspect embedding inputs, chunking, and the text stored for dense indexing before changing fusion logic.

  • Measure recall at a fixed cutoff alongside ranking-oriented measures.
  • Track result overlap between dense and sparse lists; low overlap is a useful diagnostic signal.
  • Test metadata filters and access controls before fusion, not as an afterthought.
  • Log retriever-specific ranks so relevance regressions can be traced to a source.