Why score-based blending is harder than it looks

Dense retrieval and BM25 produce scores for different reasons. A dense score reflects the relationship between vector representations, while BM25 emphasizes term matches and term-frequency statistics. Even if both systems return a numeric score, the ranges and distributions are not inherently comparable.

A fixed weighted sum can therefore behave unpredictably. One retriever may dominate simply because its score range is wider, not because its result is more useful for the query. Score normalization can help, but it introduces another set of choices and should be validated against representative queries.

  • Dense retrieval can surface paraphrases and conceptually related content.
  • BM25 is often valuable for identifiers, product names, error messages, and uncommon terms.
  • Raw scores should not be treated as interchangeable across retrieval methods.

Use Reciprocal Rank Fusion as a stable first combination method

RRF assigns a document a contribution based on its position in each ranked list. For every list where a document appears, add 1 divided by k plus its rank. The final fused score is the sum of those contributions, and documents are sorted by that total.

The constant k reduces the difference between adjacent top ranks and makes fusion less sensitive to small ranking variations. Because RRF uses position rather than raw retrieval scores, it avoids requiring S3 Vectors and Quickwit BM25 to share a common scoring scale.

  • Run dense and sparse retrieval independently for the same query.
  • Keep a bounded candidate list from each retriever.
  • Assign ranks starting at 1 within each list.
  • Compute: RRF(document) = sum of 1 / (k + rank) across lists.

Make fusion observable and tune it with real query sets

Store retrieval provenance alongside each fused result: whether it came from dense search, BM25, or both; its rank in each source; and its final RRF score. This makes it possible to diagnose why a result appeared and to identify queries where one retrieval path consistently contributes little value.

Evaluate changes with a small, curated query set before broad rollout. Include semantic questions, exact-name lookups, multi-term troubleshooting queries, and queries with ambiguous vocabulary. The goal is not to make every query depend equally on both methods, but to ensure the combined ranking improves the result set for the workload that matters.

  • Log per-source ranks and candidate counts for each query.
  • Inspect results unique to dense retrieval and unique to BM25.
  • Test different candidate depths before changing fusion behavior.
  • Treat k and any source-specific weighting as evaluation-driven configuration.