Why raw-score fusion is fragile

A dense retrieval score reflects the similarity function, embedding model, and vector representation used to produce it. A BM25 score reflects term frequency, document frequency, field length, and query terms. Even if both results are sorted correctly within their own systems, a dense score of 0.72 and a BM25 score of 8.4 are not measurements on a shared scale.

Normalizing scores can help in tightly controlled cases, but it creates another dependency: the normalization method must remain useful as corpora, queries, embeddings, analyzers, and document lengths change. Rank-based fusion starts from a smaller and more stable assumption: a result near the top of either list is useful evidence.

  • Dense search is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact terminology, codes, names, and rare phrases.
  • Raw score ranges can shift independently between retrieval methods.
  • A fusion strategy should preserve strong candidates from either retriever.

Fuse ranked lists with RRF

Reciprocal Rank Fusion assigns each document a contribution based on its position in each ranked list. For a document d, the fused score is the sum of 1 divided by k plus its rank in every list where it appears. The constant k reduces the influence of very small rank differences near the top of a list.

For example, retrieve a candidate list from S3 Vectors and another from Quickwit BM25. Assign ranks beginning at 1, merge documents by their stable document identifier, calculate the RRF score, then sort descending. A document that ranks well in both lists will rise naturally, while a document that is exceptional in only one list can still remain competitive.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use the same document identifier in dense and sparse indexes.
  • Choose candidate depths large enough to create meaningful overlap and coverage.
  • Start with a fixed k such as 60, then evaluate it against representative queries.

Make fusion observable and testable

RRF is simple enough to implement at the API layer, but it still needs retrieval evaluation. Build a small query set that includes semantic requests, exact identifiers, abbreviations, quoted phrases, and domain-specific terminology. For each query, record whether the relevant document appears in the dense list, sparse list, and fused list.

Also inspect disagreement rather than treating it as failure. If BM25 finds an exact model number that dense retrieval misses, that may be expected. If dense retrieval consistently finds useful paraphrases that BM25 misses, that is also expected. The goal of fusion is not to make both retrievers identical; it is to turn their different strengths into a stronger candidate set.

  • Log per-retriever rank alongside the fused rank for debugging.
  • Evaluate recall at the candidate-set cutoff before evaluating downstream ranking.
  • Use a fixed test set when changing embeddings, analyzers, or chunking.
  • Keep dense and sparse documents synchronized so identifiers resolve to the same content.