Why raw score blending is fragile

A tempting hybrid-search design is to add a dense similarity score to a BM25 score. That approach is often unstable because the values have different meanings and ranges. A cosine-like similarity measure and a BM25 relevance score are not naturally calibrated against one another, and their distributions can change with the corpus, query wording, analyzer settings, or embedding model.

Normalization can help, but it introduces operational choices: which historical window to use, how to handle outlier queries, and whether a score distribution has shifted. For many retrieval systems, it is safer to begin with rankings rather than raw scores. A rank says only that one candidate was ordered ahead of another within a retrieval method.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for literal matches such as IDs, names, and rare terms.
  • Raw scores from separate retrieval methods should not be assumed comparable.
  • Rank-based fusion reduces dependence on score calibration.

Fuse dense and sparse candidates with RRF

Reciprocal Rank Fusion (RRF) combines independently ranked result lists. For each document, add a small contribution from every list in which it appears: 1 divided by k plus that document's rank. The constant k dampens the advantage of a first-place result and prevents the top few positions from dominating too aggressively.

For a query, request a candidate list from dense search over S3 Vectors and another from Quickwit BM25. Join the lists using a stable document or chunk ID, calculate the RRF total, then sort by that total. A document appearing near the top of both lists will rise naturally, while a strong result from just one method can still remain competitive.

  • Use a shared, stable ID for the same chunk in dense and sparse indexes.
  • Choose a candidate depth large enough to allow overlap and useful long-tail candidates.
  • Compute: RRF(document) = Σ 1 / (k + rank).
  • Treat k and per-retriever weights as evaluation parameters, not permanent assumptions.

Make fusion observable and testable

Log more than the final fused ranking. For each selected result, retain its dense rank, BM25 rank, fused score, and which retrieval paths returned it. This makes it possible to diagnose whether a poor answer came from missing candidates, weak fusion, bad chunk boundaries, or downstream generation behavior.

Evaluate with a query set that reflects production language. Include exact-lookup queries, paraphrased questions, acronym-heavy requests, and queries that should return no result. Compare dense-only, BM25-only, and fused retrieval using a relevance judgment process appropriate to the application. Fusion is worthwhile when it improves the retrieval quality that matters for users, not merely when it changes ranking order.

  • Record candidate provenance before passing context to a generator.
  • Inspect queries where dense and BM25 strongly disagree.
  • Keep index-time chunk IDs and metadata consistent across both retrieval paths.
  • Re-run evaluation after changing embeddings, analyzers, chunking, or corpus content.