Why raw-score blending is fragile

A tempting hybrid strategy is to add a dense similarity score to a BM25 score. The problem is that the two values are produced by different retrieval models and carry different distributions. A score that is large in one system is not automatically more meaningful than a smaller score in the other.

Even after normalization, score behavior can shift as document length, corpus composition, embedding models, analyzers, and query patterns change. A fixed weighting rule can therefore become an ongoing calibration task rather than a simple retrieval feature.

Rank fusion takes a narrower and often useful view: each retriever supplies an ordered list of candidates, and the fusion layer rewards documents that appear near the top of one or both lists.

  • Use dense retrieval for semantic similarity and paraphrases.
  • Use BM25 for lexical precision, rare tokens, and exact phrase signals.
  • Treat each retriever’s rank as the common interface rather than assuming its score is portable.

Apply reciprocal rank fusion at the candidate layer

For a query, request the top K dense results from regional S3 Vectors and the top K sparse results from Quickwit BM25. Deduplicate document identifiers, then assign every candidate an RRF score: sum 1 divided by k plus its rank for each list in which it appears. Here, rank starts at 1 and k is a positive constant selected by the application.

A document ranked highly by both retrievers receives two contributions. A document returned by only one retriever can still enter the final results, which preserves the complementary value of the two search paths. The output is then sorted by the fused score before returning the top N documents.

Keep the original dense rank, BM25 rank, and fused score in retrieval logs. This makes it possible to inspect why a result surfaced without incorrectly interpreting the underlying scores as equivalent.

  • Retrieve the same candidate depth K from both paths as an initial baseline.
  • Deduplicate using a stable document or chunk identifier before scoring.
  • Compute contributions only when a candidate is present in a ranked list.
  • Return a smaller final set N after fusion, sized for the downstream reader or generator.

Make fusion debuggable before making it elaborate

Start with a small evaluation set that reflects real queries: acronym-heavy requests, natural-language questions, product names, code-like strings, and queries whose answer is expressed with different wording. For each query, compare dense-only, BM25-only, and fused ranked lists against a documented relevance judgment or an operational review process.

When results are surprising, inspect the two input lists first. A fusion method cannot recover a document that neither retriever returned in its candidate set. If relevant material is consistently absent, investigate chunking, metadata filters, embedding inputs, sparse-field construction, or candidate depth before changing fusion math.

RRF is deliberately simple, which makes it a useful baseline rather than a claim that one ranking recipe fits every corpus. Once the baseline is observable, teams can test query routing, metadata-aware retrieval, or learned reranking with a clear point of comparison.

  • Log query text or a privacy-safe query representation alongside result ranks.
  • Record document IDs and source path for every fused candidate.
  • Review zero-result and no-click queries separately from ordinary ranking errors.
  • Change one retrieval variable at a time so evaluation remains interpretable.