Why raw-score blending is fragile

Dense retrieval typically returns a similarity or distance-derived score based on embedding vectors. BM25 returns a lexical relevance score based on term occurrence and document statistics. Even when both lists are sorted correctly on their own, the numeric ranges and distributions behind those scores can differ substantially.

Adding or averaging raw scores assumes that a value from one retriever has the same meaning as a value from the other. That assumption can break as corpus composition, query wording, analyzers, embedding models, or retrieval settings change. The result is a hybrid ranker whose behavior is difficult to reason about.

  • A high BM25 score is not inherently equivalent to a high dense-similarity score.
  • Score ranges can vary between queries, not only between retrieval systems.
  • Changing one retriever can unexpectedly alter the balance of a score-based blend.

Fuse ordered lists with RRF

RRF assigns each document a contribution based on its position in each result list. For a document d, sum 1 divided by k plus its rank for every list where it appears. The constant k reduces the difference between adjacent top ranks and prevents a single first-place placement from dominating too aggressively.

In a Talqora-style hybrid flow, retrieve a candidate list from regional S3 Vectors and another from Quickwit BM25, then merge document identifiers and calculate an RRF score. Sort the merged candidates by that score before returning the final shortlist or passing it to a later reranking step.

  • RRF(d) = sum over result lists of 1 / (k + rank(d)).
  • Use one-based ranks: the first result has rank 1.
  • Choose the same candidate depth for both retrievers initially, then evaluate changes deliberately.
  • Deduplicate by a stable document or chunk identifier before producing the fused list.

Treat fusion as a controlled retrieval stage

RRF is useful because it rewards agreement while still preserving documents that one retrieval method finds uniquely valuable. A chunk that ranks well in both dense and sparse retrieval tends to move upward. A chunk found only by BM25 can still survive when exact terminology matters, while a semantically related chunk can survive when vocabulary differs.

Keep fusion observable. Record each candidate's dense rank, BM25 rank, and fused rank during evaluation. Reviewing these fields for representative queries makes it easier to identify whether misses come from chunking, indexing, query construction, candidate depth, or the fusion rule itself.

  • Start with a fixed k and document it alongside retrieval configuration.
  • Evaluate queries containing exact identifiers, paraphrases, abbreviations, and multi-concept questions.
  • Inspect cases where only one retriever returned the selected document.
  • If a reranker is used later, retain retrieval-rank metadata for debugging.