Why raw-score blending is fragile

A tempting hybrid strategy is to request a dense ranking and a BM25 ranking, normalize both score arrays, then add them with a chosen weight. The problem is that the scores originate from different retrieval models and often have different distributions. A BM25 score is influenced by term frequency and corpus statistics, while a dense similarity score reflects vector geometry.

Even when both scores are normalized per query, the result can be sensitive to outliers, result-set depth, filters, and changes in either index. A dense score of 0.72 does not carry a universal meaning that can safely be added to a BM25 score of 8.4. Rank-based fusion sidesteps that assumption.

  • Do not assume sparse and dense scores have a shared scale.
  • Avoid using a single global weight before inspecting query behavior.
  • Treat rankings as the stable common interface between retrievers.

Apply reciprocal rank fusion to two candidate lists

For each query, retrieve a candidate list from Talqora's dense path backed by regional S3 Vectors and a second list from the sparse path backed by Quickwit BM25. Keep document identifiers consistent across both paths. Then assign each document an RRF score based on its rank in each list: RRF(d) = Σ 1 / (k + rank_i(d)).

The constant k reduces the influence of the very top positions and is commonly set to a moderate value such as 60. A document appearing near the top of both lists rises naturally. A document found by only one retriever can still make the final list, which preserves the complementary recall of dense and lexical search.

  • Fetch the same candidate depth from both retrievers as an initial baseline.
  • Use one canonical document ID for deduplication before scoring.
  • Assign ranks starting at 1, not 0.
  • Sort by fused score and retain the top results for presentation or reranking.

Evaluate fusion with query slices, not only averages

RRF has few moving parts, but it still needs evaluation. Build a test set that reflects the traffic your application expects. Include identifier-heavy queries, short natural-language questions, product terminology, misspellings where relevant, and queries whose answer depends on phrasing rather than exact token overlap.

Compare dense-only, BM25-only, and fused rankings using the same relevance judgments. Review failures by query type. If exact identifiers are consistently missing, increase sparse candidate depth or verify tokenization and indexing. If semantically related material is absent, inspect embedding generation, chunk boundaries, and dense candidate depth before changing fusion parameters.

  • Measure recall-oriented metrics at the candidate cutoff and ranking metrics at the displayed cutoff.
  • Log which retriever contributed each final document.
  • Test filters and tenant boundaries as part of retrieval evaluation.
  • Change one variable at a time: candidate depth, k, or upstream indexing.