Why raw dense and BM25 scores should not be added

A dense-search score and a BM25 score are produced by different retrieval models and have different distributions. Their numeric ranges can change with the query, corpus, analyzer settings, vector distance choice, or implementation details. A weighting rule such as `0.5 × dense_score + 0.5 × bm25_score` can therefore give one retriever unintended control over the final ranking.

Score normalization can sometimes help, but it introduces another system to tune and monitor. It may require collecting score statistics, selecting a normalization method, and checking whether that method remains appropriate as documents and query traffic change. For a first hybrid ranking implementation, use the information both systems reliably provide: rank order.

  • Dense retrieval can surface paraphrases and conceptually related content.
  • BM25 can reward exact terms, identifiers, product names, and rare phrases.
  • Raw score magnitude is not a universal measure of relevance across retrievers.
  • Rank positions are directly usable even when score scales differ.

Fuse two ranked lists with RRF

Reciprocal Rank Fusion assigns each document a contribution based on its position in each result list. For every list where a document appears, add `1 / (k + rank)`. Here, rank starts at 1 and `k` is a positive constant that reduces the difference between nearby positions. Sort documents by their summed contribution.

In an application using Talqora’s dense and sparse retrieval paths, request a candidate list from regional S3 Vectors and another from Quickwit BM25. Deduplicate by a stable document or chunk identifier, calculate the RRF total, and return the highest-ranked merged results. A document found by both methods receives contributions from both lists, while a strong one-method result can still remain competitive.

  • Formula: `RRF(document) = Σ 1 / (k + rank_i)`.
  • Use the same stable identifier in dense and sparse indexes so matching results can be merged.
  • Choose candidate depths large enough to create useful overlap and alternatives.
  • Keep the original dense and BM25 ranks in logs for debugging and evaluation.

Make fusion observable before making it complicated

Start with a small labeled query set that reflects real retrieval needs: exact identifiers, short keyword queries, natural-language questions, and terminology with synonyms. Compare dense-only, BM25-only, and fused results. Review not only whether the desired document appears, but whether it appears early enough for the consuming experience.

Instrument the merged response with provenance. For each result, record whether it came from dense retrieval, BM25 retrieval, or both, along with its source ranks and final RRF score. This makes failures actionable: a missing result may indicate indexing coverage, a weak sparse query, an embedding mismatch, insufficient candidate depth, or a fusion setting that needs review.

  • Evaluate result quality by query class, not only by an aggregate average.
  • Inspect one-source and two-source results separately.
  • Treat `k` and candidate depth as explicit configuration values.
  • Change one variable at a time when reviewing relevance results.