Why raw-score blending is risky

Dense retrieval ranks documents by proximity between embeddings, while BM25 ranks documents using term frequency, document statistics, and query terms. Even when both systems return numeric scores, those values represent different calculations and can vary with query shape, corpus changes, and implementation details.

Adding or averaging raw scores can therefore make one retrieval method dominate for reasons unrelated to relevance. A query containing a rare product code may benefit from BM25, while a question phrased in unfamiliar language may benefit from dense retrieval. Rank-based fusion lets each retriever contribute without requiring a shared score scale.

  • Use dense retrieval for semantic similarity and paraphrased intent.
  • Use BM25 for exact terms, identifiers, and uncommon vocabulary.
  • Treat scores from separate retrieval methods as method-specific signals.
  • Fuse ordered result lists rather than assuming score values are interchangeable.

Apply reciprocal rank fusion

RRF assigns each document a contribution based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the difference between adjacent high ranks and prevents a single first-place result from overwhelming all other evidence.

In a Talqora-oriented retrieval path, an application can request a dense candidate list from regional S3 Vectors and a sparse candidate list from Quickwit BM25. It can then merge document identifiers, calculate RRF scores, and return the highest-ranked unique documents. This keeps fusion logic explicit at the API consumer or retrieval-service layer.

  • Request a candidate set from both dense and sparse retrieval paths.
  • Use stable document IDs to identify duplicates across result lists.
  • Assign ranks starting at 1, not 0.
  • Choose and document a fixed k value before evaluating changes.

Evaluate the retrieval pipeline, not just the formula

RRF is deliberately simple, but its quality depends on the candidate lists it receives. If a relevant document never appears in either candidate set, fusion cannot recover it. Candidate depth, chunking choices, metadata filters, and the quality of source content all affect the final result set.

Build a small evaluation set from real search tasks, including exact-name queries, ambiguous questions, paraphrases, and multi-concept requests. Compare dense-only, BM25-only, and fused rankings using relevance judgments. Review failures by query type so adjustments are tied to observed behavior rather than intuition.

  • Keep candidate depths consistent while comparing retrieval strategies.
  • Measure whether relevant documents appear near the top of each result set.
  • Inspect duplicate chunks and apply diversification when needed.
  • Log query type, retriever ranks, and final fused rank for debugging.