Why raw-score blending is fragile

A tempting hybrid strategy is to add a dense-search score to a BM25 score. That is usually unreliable because the two values are not inherently comparable. Their ranges, distributions, and sensitivity to query length can differ, and a change in either retrieval system can alter the meaning of a score.

Rank-based fusion avoids asking whether a dense score of one value is equivalent to a BM25 score of another. Instead, it asks a simpler question: which documents appear near the top of either candidate list? This makes fusion easier to reason about when dense and sparse retrieval are operated as separate searches.

  • Dense retrieval can help with paraphrases and conceptual similarity.
  • BM25 can preserve exact matching for product names, codes, and uncommon terms.
  • Raw scores should not be assumed to share a common scale.
  • Rank positions are directly usable across both result lists.

Apply reciprocal rank fusion to two candidate lists

Run a dense query against S3 Vectors and a BM25 query against Quickwit using the same user query and the same document identifiers. Request a bounded candidate set from each system, then merge results by document ID. For every occurrence of a document at rank r, add 1 divided by k plus r to its fused score.

The formula is RRF(d) = Σ 1 / (k + rank_i(d)), where the sum covers each retrieval list containing document d. The constant k reduces the difference between nearby ranks and prevents the first position from dominating too sharply. A commonly used starting point is 60, but it is a parameter to validate against representative queries rather than a universal default.

  • Use one stable document ID in both the vector and BM25 indexes.
  • Rank each list starting at 1, not 0.
  • Deduplicate by document ID before returning results.
  • Sort documents by descending fused score and retain the desired final count.

Keep the fusion layer observable and testable

Treat RRF as a small, explicit application-layer ranking step. Log which retriever contributed to each returned document, its rank in each list, and the final fused score. This makes it possible to inspect whether exact-match queries are receiving useful BM25 support and whether semantic queries are benefiting from dense retrieval.

Evaluate with a query set drawn from real retrieval tasks, including terminology-heavy queries, natural-language questions, and queries containing identifiers. Compare dense-only, BM25-only, and fused rankings using human judgments or task-specific relevance labels. If results are weak, first inspect indexing consistency and candidate depth before changing fusion parameters.

  • Record dense rank, BM25 rank, and fused rank for returned documents.
  • Check that document IDs and deletion behavior remain consistent across indexes.
  • Test candidate-list depth separately from the final number of returned results.
  • Use relevance judgments to choose parameters and regressions to catch ranking changes.