Why raw-score blending is brittle

A dense retrieval score and a BM25 score are produced by different models and have different distributions. A larger value in one system does not automatically represent the same degree of relevance as a larger value in the other. Even within one retrieval method, score ranges may shift as the corpus, query mix, analyzer settings, or embedding model changes.

A weighted formula such as `dense_score + sparse_score` can therefore appear to work on a small test set and then degrade when query language changes or new content arrives. Before tuning weights, a retrieval pipeline needs a common unit that does not assume score comparability. Rank is that unit: each retriever can state its ordered preference even when its numeric scale differs.

  • Keep dense and sparse scores available for debugging, but do not assume they are interchangeable.
  • Request enough candidates from each retriever to create a useful fusion pool.
  • Use a stable document or chunk identifier so results from both lists can be deduplicated.
  • Evaluate exact-term, semantic, and mixed-intent queries separately.

Fuse the rankings with RRF

RRF assigns each document a contribution based on its position in each result list. For a document at rank `r`, the contribution is `1 / (k + r)`, where `k` is a positive constant. Sum the contributions across lists, then sort documents by the total. A result that ranks well in both dense and BM25 retrieval rises naturally, while a strong result from only one retriever can still survive.

The constant `k` reduces the difference between adjacent top ranks. It is not a universal relevance setting; it controls how sharply the fusion favors early positions. Start with one documented value, use one-based ranks consistently, and change it only after reviewing labeled or human-judged queries. The important property is that RRF consumes ordering, not incompatible raw scores.

  • For each query, retrieve a ranked dense list from S3 Vectors and a ranked sparse list from Quickwit BM25.
  • For every returned ID, add `1 / (k + rank)` for each list in which it appears.
  • Deduplicate IDs before returning the final ranking.
  • Apply metadata filters consistently before fusion when the same filter semantics are required by both retrieval paths.

Make fusion observable before making it clever

A hybrid ranking is easier to improve when each result records how it got there. Log the fused rank, dense rank when present, sparse rank when present, and the retrieval branch or branches that returned the item. This makes it possible to distinguish a BM25-dominant exact-match query from a dense-dominant conceptual query without inferring the explanation from one final score.

Start with a small query set that represents real traffic: acronym-heavy queries, IDs and names, paraphrases, broad topical questions, and queries with restrictive filters. Review the top results and failure cases. If one retrieval branch repeatedly contributes irrelevant candidates, investigate its input processing or candidate depth before introducing complicated score calibration or learned rerankers.

  • Track overlap between dense and sparse candidate lists; low overlap is not automatically bad, but it is informative.
  • Inspect the source ranks of clicked, selected, or manually relevant results.
  • Version the embedding model, BM25 configuration, fusion constant, and candidate depths alongside evaluation results.
  • Treat RRF as a transparent baseline that can remain in place while later ranking stages are tested.