Why raw score blending is fragile

It is tempting to request a dense result set and a BM25 result set, then add their scores together. That approach is often unreliable because the two systems produce scores with different meanings and distributions. A dense similarity score and a BM25 relevance score are not automatically comparable.

Score ranges can also shift as collections, queries, analyzers, embeddings, or retrieval settings change. A weighting scheme that appears sensible for one set of queries may over-favor one retriever for another. Rank-based fusion avoids depending on a shared score interpretation.

  • Dense retrieval helps when a query and a document express the same idea differently.
  • BM25 helps when literal wording, product codes, names, or error messages are important.
  • A document need not receive a similar numeric score from both systems to be useful.

Fuse ranked lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in each ranked list. For every occurrence of a document, add 1 divided by k plus its rank. Then sort documents by their accumulated total. Here, rank starts at 1 and k is a positive constant chosen to reduce the impact of small rank differences near the top.

For example, retrieve a bounded candidate list from S3 Vectors for the dense query and another from Quickwit BM25 for the sparse query. If a document ranks highly in either list, it receives a strong contribution. If it appears in both lists, its contributions add together. The method works even when the original scores are unavailable or intentionally ignored.

  • RRF score: sum of 1 / (k + rank) across result lists.
  • Use stable document IDs to identify the same item across dense and sparse results.
  • Deduplicate before returning results to the caller.
  • Keep the original dense and BM25 ranks for debugging and evaluation.

Make the fusion layer observable and testable

Treat fusion as a small, explicit retrieval component rather than an opaque post-processing step. Log the query, candidate counts, source ranks, fused rank, and document ID for sampled requests. These records make it possible to diagnose whether a poor result came from embedding retrieval, BM25 retrieval, missing candidates, or the fusion rule.

Evaluate with a representative query set that includes conceptual questions, exact-title lookups, identifiers, abbreviations, and multi-term queries. Review not only whether a relevant document appears, but where it appears in the fused list. Adjust candidate depths and the RRF constant deliberately, using relevance judgments rather than a single anecdotal query.

  • Fetch enough candidates from each retriever for useful overlap and recovery.
  • Use consistent filters and access controls before fusing results.
  • Record source-specific ranks to investigate ranking regressions.
  • Re-run an evaluation set whenever embeddings, analyzers, or indexed content change.