Why raw-score blending is fragile

A dense retrieval score and a BM25 score are produced by different models and ranking functions. Even when both are useful signals, a score of 0.8 from one retriever does not have an inherent numerical relationship to a score of 0.8 from the other.

A weighted formula such as `dense_score + bm25_score` therefore needs careful normalization and ongoing tuning. Score distributions can also shift when embeddings, analyzers, document collections, or query patterns change. Rank-based fusion avoids assuming that either system's score scale is portable.

  • Dense retrieval helps with semantic similarity and paraphrased queries.
  • BM25 helps preserve exact-term, acronym, and identifier matches.
  • Rank positions are easier to compare across independent retrievers than raw scores.

Apply reciprocal rank fusion to two candidate lists

Run the same user query through the dense retrieval path backed by regional S3 Vectors and the sparse path backed by Quickwit BM25. Request a candidate window from each path, deduplicate documents by a stable document ID, and assign each document an RRF contribution for every list in which it appears.

For a document at one-based rank `r`, RRF adds `1 / (k + r)`. The final score is the sum across lists. The constant `k` reduces the difference between nearby ranks near the top of a list; it is a tuning parameter, not a universal truth. Sort by the fused score and return the top results.

  • Use one-based ranks: the first result has `r = 1`.
  • Deduplicate before presenting results, while retaining contributions from both lists.
  • Keep retrieval provenance, such as `dense`, `bm25`, or `both`, for debugging.
  • Choose candidate-window size and `k` through evaluation on representative queries.

Make fusion observable and test it by query class

Hybrid retrieval should be evaluated against the queries people actually ask. Build a small judged set that includes exact identifier lookups, terminology-heavy questions, paraphrases, and broad conceptual questions. Compare dense-only, BM25-only, and fused rankings using the same candidate and relevance rules.

Log enough information to explain ranking outcomes: query text or an approved representation of it, returned document IDs, rank in each source list, fused rank, and source provenance. These records reveal whether fusion is rescuing useful results or merely increasing candidate overlap.

  • Inspect failures separately for exact-match and semantic query classes.
  • Watch for duplicate or near-duplicate documents dominating both lists.
  • Version embedding choices, BM25 configuration, fusion parameters, and evaluation sets.
  • Treat RRF as a clear baseline before adding more complex learned ranking.