Why raw-score blending is fragile

A common first implementation queries a dense index and a BM25 index, then adds the two returned scores with chosen weights. That can work only when both score distributions are understood, stable, and calibrated for the same corpus and query traffic. In practice, those conditions are easy to lose as embeddings, analyzers, document lengths, or index settings change.

Rank is a more portable signal. If a document is near the top of both lists, it is likely to be a useful candidate even when the dense and sparse numeric scores have unrelated meanings. RRF turns that observation into a simple, deterministic merge step.

  • Do not assume cosine-like dense scores and BM25 scores are on a shared scale.
  • Avoid making fixed score thresholds serve both retrieval systems.
  • Treat each retriever as an independently ranked candidate generator.
  • Keep each source's rank and score in logs for later diagnosis.

Merge candidates with reciprocal rank fusion

For each query, request a bounded candidate list from S3 Vectors and a bounded candidate list from Quickwit BM25. Deduplicate documents by a stable document identifier. For every list in which a document appears, add 1 divided by k plus that document's one-based rank. The document's fused score is the sum of those contributions.

The constant k softens the difference between adjacent positions near the top of a list. A document ranked first receives somewhat more credit than one ranked second, but the merge does not let one small rank difference dominate the entire result. Choose k as a configuration value, record it with your experiment, and evaluate it against representative queries rather than treating one value as universal.

  • Dense rank 1 contributes: 1 / (k + 1).
  • Sparse rank 3 contributes: 1 / (k + 3).
  • A document in both lists receives both contributions.
  • A document absent from a list receives no contribution from that list.

Make fusion observable and testable

Store retrieval traces that show the dense rank, BM25 rank, fused rank, and final document identifier for sampled queries. This makes surprising results explainable: an exact-match document may be promoted by BM25, while a terminology mismatch may be recovered by dense search. It also reveals duplicate IDs, stale metadata, and candidate lists that are too shallow.

Evaluate with a query set that reflects actual retrieval work. Include exact identifiers, product names, short ambiguous requests, paraphrases, and multi-concept questions. Compare dense-only, sparse-only, and fused rankings using the relevance judgments available to your team. The goal is not to prove that fusion always wins; it is to understand which query classes benefit and where tuning is needed.

  • Use the same document IDs across dense and sparse indexes.
  • Start with equal source participation before adding source-specific weights.
  • Inspect failures by query type, not only aggregate metrics.
  • Version the embedding model, text processing, candidate depth, and fusion settings together.