Why score blending is often fragile

A dense retriever and a BM25 retriever typically emit scores with different meanings and ranges. A cosine-like dense similarity value is not inherently equivalent to a BM25 relevance score. Adding the two values together can make rankings sensitive to changes in embedding models, corpus composition, analyzers, or BM25 parameters.

Min-max normalization or other score-calibration methods can be useful when they are carefully evaluated, but they introduce assumptions about score distributions. For an initial hybrid retrieval design, it is often safer to combine rankings rather than raw scores.

  • Dense retrieval helps with semantic similarity and alternate phrasing.
  • BM25 helps preserve exact-token and rare-term matches.
  • Raw score scales should not be assumed comparable.
  • Rank-based fusion creates a stable starting point for evaluation.

Merge candidate lists with reciprocal rank fusion

RRF assigns each document a fused score based on its rank in each result list. For every retriever that returns a document, add 1 divided by k plus that document's rank. The constant k reduces the difference between adjacent high ranks and keeps one list from dominating merely because of a small rank change.

For example, retrieve the top N dense candidates from Talqora's regional S3 Vectors-backed dense search and the top N sparse candidates from its Quickwit BM25-backed sparse search. Deduplicate by document identifier, calculate the RRF score for every candidate, and sort descending. A document present in both lists receives contributions from both rankings.

  • Use a stable document ID to deduplicate candidates.
  • Treat ranks as one-based: rank 1 is the first result.
  • A common starting formula is RRF(d) = Σ 1 / (k + rank_i(d)).
  • Keep the dense and sparse candidate counts explicit configuration values.

Evaluate retrieval behavior before tuning

Start with a representative query set that includes exact identifiers, product names, natural-language questions, abbreviations, and queries with ambiguous wording. Record whether the expected document appears in the candidate set and where it ranks. This reveals whether dense retrieval, BM25, or their combination is contributing useful recall.

Then tune one variable at a time: candidate depth, the RRF constant, and any field or query construction choices in the sparse retriever. Inspect failures rather than only aggregate metrics. If exact IDs are missed, sparse query handling may need attention; if paraphrases are missed, the embedding or document chunking strategy may be the more relevant area to investigate.

  • Include both known-item and exploratory queries in the test set.
  • Compare dense-only, BM25-only, and fused rankings.
  • Log source ranks alongside the final fused rank for debugging.
  • Re-run the same evaluation after corpus, chunking, or embedding changes.