Why raw-score blending is fragile

Dense retrieval and BM25 answer different matching questions. Dense search ranks content by semantic proximity in an embedding space, while BM25 rewards overlap between query terms and document terms. Both signals can be useful for the same query, but their numeric outputs do not inherently mean the same thing.

A weighted formula such as dense_score + bm25_score can therefore be unstable. A change in embedding model, corpus composition, BM25 configuration, or search implementation can alter score distributions and silently change which signal dominates. The result may look like a relevance regression even when each individual retriever remains useful.

  • Dense scores depend on the embedding representation and similarity measure.
  • BM25 scores depend on term statistics, document length, and query terms.
  • A fixed weight can become inappropriate when either retrieval system changes.
  • Rank position is often more portable than a raw relevance score.

Build two candidate lists before choosing a winner

Run dense retrieval against the vector collection and BM25 retrieval against the sparse index for the same normalized query. Keep the document identifier and rank from each result list, then form a union of identifiers. This produces a candidate set containing semantic matches, exact-term matches, and documents found by both methods.

For many applications, retrieve more candidates from each source than will be returned to the user. The right depth depends on the corpus and query mix, so it should be evaluated with representative queries rather than assumed. The important design point is that fusion operates on a shared candidate set, not on whichever retriever happened to return first.

  • Use stable document or chunk IDs in both retrieval paths.
  • Apply the same filtering and access-control constraints to both paths.
  • Record each candidate's dense rank and BM25 rank.
  • Deduplicate the union before producing the final ranking.

Use reciprocal rank fusion as a robust baseline

Reciprocal rank fusion, commonly abbreviated RRF, assigns a candidate a contribution based on its position in each list. A simple form is score(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the candidate's rank in retrieval list i and k is a positive constant. Documents present in both lists receive two contributions; documents that rank highly in either list can still surface.

RRF does not require dense and BM25 scores to be calibrated against one another. It is not a substitute for evaluation, and it will not solve poor chunking, missing metadata filters, or irrelevant source material. It is, however, a clear baseline for Talqora workloads that need to combine S3 Vectors dense candidates with Quickwit BM25 candidates while avoiding arbitrary score arithmetic.

  • Start with rank-based fusion before introducing score normalization.
  • Evaluate dense-only, BM25-only, and fused rankings on the same query set.
  • Inspect failures by query type: identifiers, quoted text, conceptual questions, and mixed queries.
  • Tune candidate depths and the RRF constant only after establishing a repeatable evaluation set.