Why raw dense and BM25 scores should not be compared directly

A dense-search score reflects the similarity function and embedding model used to represent a query and document. A BM25 score reflects term frequency, document frequency, field length, and the BM25 parameters selected for an index. Even when both systems return larger-is-better scores, the numeric values do not share a natural unit.

Normalizing each list can sometimes help, but normalization adds assumptions about score distributions and can behave unpredictably when a query has a weak or unusually strong set of matches. Ranking-based fusion avoids this calibration problem by using the information both systems reliably provide: result order.

  • Dense retrieval can surface semantically related documents that use different wording.
  • BM25 can prioritize exact identifiers, error codes, names, and rare terms.
  • A score of 0.8 in one retrieval system is not inherently comparable to 0.8 in another.
  • Rank positions are easier to combine across independently tuned retrieval systems.

Apply reciprocal rank fusion to two candidate lists

Run the same query through dense search in regional S3 Vectors and sparse search through Quickwit BM25. Request a bounded candidate list from each system, then join results using a stable document identifier. For every document appearing in either list, add a contribution of 1 divided by k plus its rank in each list.

The fused score can be written as RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in retrieval list i. The constant k reduces the gap between very high ranks and lower ranks. A commonly used starting value is 60, but it should be treated as a tuning choice rather than a universal setting.

  • Use one canonical document ID in both the dense and sparse indexes.
  • Treat a missing document from a list as contributing zero from that list.
  • Deduplicate candidate IDs before presenting final results.
  • Keep the original dense rank and BM25 rank for debugging and evaluation.

Make fusion observable before making it more complex

Start with equal treatment for the dense and sparse lists. Then inspect queries where the top fused result came from only one retriever, from both retrievers, or from conflicting ranks. These groups reveal whether a corpus is driven primarily by exact vocabulary, semantic paraphrase, or a mixture of both.

If one retriever consistently contributes noisy candidates for a known query class, address the cause before adding elaborate weighting. Examples include improving document chunk boundaries, indexing a more appropriate text field for BM25, or reviewing how identifiers and metadata are represented in the text sent to the embedding model.

  • Log query ID, candidate ID, source ranks, and final fused rank.
  • Evaluate a representative set containing natural-language questions and exact-term queries.
  • Change one variable at a time: candidate depth, k, or retrieval weighting.
  • Keep fusion deterministic so relevance changes can be traced to an index or configuration change.