Why raw score blending is a fragile default

Dense retrieval and BM25 produce scores with different meanings and distributions. A dense-search score represents a vector similarity relationship, while BM25 is driven by term frequency, document frequency, and document length. Adding those values directly assumes they share a stable scale, which is usually not a safe assumption.

Score distributions can also shift as a corpus grows, chunking changes, embeddings are replaced, or text analysis settings evolve. A weight that appears reasonable on one evaluation set may become misleading after an indexing or content change. Rank-based fusion avoids requiring those raw values to be calibrated against each other.

  • Use dense retrieval to capture semantic matches and paraphrases.
  • Use BM25 to preserve exact-match behavior for terms and identifiers.
  • Treat dense and sparse scores as source-specific signals, not interchangeable numbers.

Fuse ranked lists with RRF

Run the same query through the dense path backed by regional S3 Vectors and the sparse path backed by Quickwit BM25. Each path returns an ordered list of document or chunk identifiers. For every unique result, assign an RRF score based on its position in each list.

The standard formula is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in result list i and k is a positive constant. A document appearing near the top of both lists receives a strong combined score; a document that appears in only one list can still be retained.

  • Fetch a candidate window from both retrieval paths before fusion.
  • Deduplicate by a stable document or chunk identifier.
  • Sum the reciprocal-rank contribution from every list containing the candidate.
  • Sort candidates by the resulting RRF score and return the desired top results.

Make fusion observable before tuning it

Begin with equal treatment for the dense and sparse lists, then inspect real queries rather than immediately adding weights. Build a small evaluation set that includes exact identifiers, short keyword queries, natural-language questions, ambiguous terms, and vocabulary that differs from the wording in the source material.

For each query, record the separate dense and BM25 ranks alongside the fused rank. This makes failures diagnosable: an exact-match document may be absent from sparse retrieval because of analysis or indexing, while a semantically relevant result may be missing from dense retrieval because the chunk lacks enough context. Fusion can combine candidates, but it cannot recover a document that neither path retrieves.

  • Log candidate IDs, per-source ranks, and the final fused rank.
  • Measure whether relevant results appear in either candidate list before evaluating fusion.
  • Review chunk boundaries and metadata filters when relevant items are consistently absent.
  • Introduce source weights only after evaluation shows a repeatable imbalance.