Why raw-score blending is fragile

A dense retriever and a BM25 retriever produce scores with different meanings. A dense score reflects the similarity behavior of an embedding model and its distance or similarity function. A BM25 score reflects term statistics, document length normalization, and query-term matches. Equal-looking numbers from the two systems do not necessarily represent equal evidence.

Adding or averaging raw scores can therefore make ranking sensitive to changes that are not obviously relevance changes. A new embedding model, a different corpus distribution, or a BM25 configuration adjustment may alter score ranges. Rank-based fusion instead uses an output both retrievers naturally provide: the ordering of their top results.

  • Use dense retrieval for semantic alternatives and paraphrases.
  • Use BM25 for exact tokens, rare vocabulary, codes, and literal phrasing.
  • Avoid assuming dense and sparse scores are calibrated to one shared scale.

Fuse top-k lists with reciprocal rank fusion

RRF assigns each document a contribution based on its rank in each result list. For a document d, the fused score is the sum of 1 divided by k plus its rank: RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs across the retrievers in which the document appears. Lower rank numbers contribute more, so a document that appears near the top of either list receives useful credit.

The constant k softens the difference between adjacent ranks. A commonly used starting value is 60, but it is a tuning parameter rather than a universal rule. Retrieve a bounded candidate list from regional S3 Vectors and a bounded candidate list from Quickwit BM25, deduplicate by a stable document or chunk ID, calculate the fused score, then sort descending.

  • Keep ranks one-based: the first result has rank 1.
  • Choose the same initial candidate depth for both retrievers when comparing behavior.
  • Deduplicate before returning results, while preserving contributions from every list.
  • Treat absent documents as contributing zero from that retriever.

Tune for query behavior, not a single aggregate metric

Start with a small evaluation set that reflects real traffic. Include semantic questions, identifier-heavy queries, short queries, multi-term queries, and queries with terminology that appears exactly in the corpus. For each query, record whether a useful result appears in the first few fused positions and whether the fusion improves, preserves, or harms the strongest individual retriever.

Review failures by category. If exact identifiers fall behind broad semantic matches, inspect sparse candidate depth and whether tokenization preserves the identifier. If paraphrases are missing, inspect the dense candidate depth and embedding quality. RRF is deliberately simple: it gives a stable baseline before adding query classifiers, learned rerankers, or specialized routing.

  • Log the dense rank, BM25 rank, and fused rank for returned documents.
  • Evaluate candidate depth separately from final result count.
  • Test changes against a fixed query set before changing fusion parameters.
  • Keep a per-query view; aggregate metrics can hide important regressions.