Why raw-score merging is usually a mistake

A dense search score represents similarity in an embedding space. A BM25 score reflects term occurrence, term rarity, and document-length effects. Even when both systems return numeric scores, a dense score of 0.8 and a BM25 score of 8.0 do not have a shared interpretation.

Adding, averaging, or thresholding those raw values creates a hidden calibration problem. The result can change substantially across queries, collections, embedding models, analyzers, or index settings. Ranking positions are a safer common unit because each retrieval path already expresses an ordering of its own candidates.

  • Use dense retrieval for semantic matches and paraphrases.
  • Use BM25 for exact terms, identifiers, rare names, and quoted language.
  • Keep each retrieval path independently ranked before combining results.

Fuse the two ranked lists with RRF

Reciprocal rank fusion assigns each document a contribution based on its position in each result list. For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is its one-based rank in list i and k is a positive constant. Documents found by both retrieval methods receive contributions from both lists.

A commonly used starting value for k is 60, but it is a tuning parameter rather than a universal rule. Larger values reduce the difference between nearby ranks; smaller values place more emphasis on being near the top of a list. Start with a fixed value, inspect representative queries, and change it only with a clear evaluation method.

  • Request a top-N list from S3 Vectors for the dense query.
  • Request a top-N list from Quickwit BM25 for the same user query.
  • Deduplicate by a stable document or chunk identifier.
  • Sum each document's reciprocal-rank contributions and sort descending.

Make fusion reliable in production

Hybrid fusion depends on identity consistency. The dense and sparse indexes must refer to the same logical retrieval unit: for example, the same chunk ID, document version, tenant boundary, and access policy. If one path indexes whole documents while the other indexes chunks, the fused ranking can be hard to explain and difficult to use downstream.

Treat RRF as a retrieval-stage operation, not a substitute for evaluation. Build a small query set that includes exact identifiers, natural-language questions, ambiguous terms, and domain-specific vocabulary. Compare dense-only, BM25-only, and fused top results, then inspect failures such as stale documents, duplicate chunks, or mismatched filters.

  • Apply the same tenant, authorization, and freshness filters to both paths.
  • Store stable IDs and version metadata with indexed content.
  • Log which retrieval path contributed to each fused result.
  • Evaluate candidate depth N separately from the number of results shown to users.