Why raw score mixing is fragile

Dense retrieval scores and BM25 scores answer different questions. A dense embedding score reflects proximity in an embedding space, while BM25 emphasizes term overlap and term rarity. Their numeric ranges, distributions, and sensitivity to query length are not naturally comparable.

Adding or averaging raw scores can therefore make ranking behavior difficult to predict. A query containing a rare product code may benefit strongly from BM25, while a natural-language question with paraphrased wording may benefit from dense retrieval. A fusion method should preserve useful signals from both systems without assuming that one score of a particular value means the same thing in the other system.

  • Dense retrieval helps with semantic similarity and paraphrases.
  • BM25 helps with exact terms, identifiers, and uncommon vocabulary.
  • Score normalization can be query-sensitive and operationally brittle.

Fuse ranked lists with RRF

Reciprocal Rank Fusion combines results by position rather than by raw score. Retrieve a candidate list from S3 Vectors and another from Quickwit BM25, then assign each document an RRF score based on its rank in each list. Documents that appear near the top of either list, or reasonably high in both, accumulate more evidence.

For a document d, calculate: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken across retrieval lists in which d appears. The constant k reduces the impact of tiny rank differences at the top of a list; a commonly used starting value is 60, but it should be treated as a tuning parameter for the application rather than a universal rule.

  • Request the same candidate depth from each retriever to begin with, such as the top 50 or top 100.
  • Deduplicate by a stable document or chunk identifier before producing the final order.
  • Use one-based ranks: the first result has rank 1.
  • Keep source ranks during logging so fused results remain diagnosable.

Evaluate fusion on the queries people actually ask

RRF is easy to implement, but it still needs evaluation. Build a small query set that includes natural-language questions, acronym-heavy searches, quoted phrases, SKU or ticket lookups, and queries with misspellings or alternate terminology. For each query, record whether the needed document or passage appears within the retrieval depth used by the downstream system.

Review disagreements as well as aggregate outcomes. If BM25 repeatedly finds the authoritative item for identifier queries, ensure its candidate depth is sufficient. If dense retrieval surfaces conceptually related passages with mismatched terminology, preserve that contribution. The objective is not to make the two lists look alike; it is to increase the chance that the final candidate set contains the right evidence.

  • Compare dense-only, BM25-only, and RRF-fused result sets on the same judgments.
  • Measure recall at the candidate cutoff before evaluating any later reranking step.
  • Inspect queries where a relevant result appears in only one retrieval source.
  • Version query sets and relevance judgments as the corpus changes.