Why raw-score blending is fragile

Dense retrieval ranks documents by proximity between embeddings, while BM25 ranks them from term-frequency and document-statistics signals. Even when both systems return numeric scores, those values have different meanings and ranges.

A weighted expression such as `0.7 × dense_score + 0.3 × bm25_score` therefore requires score normalization and ongoing tuning. Changes to embedding models, corpus composition, analyzers, or query behavior can alter score distributions and make previously chosen weights less useful.

  • Dense scores reflect a vector similarity or distance measure.
  • BM25 scores reflect lexical evidence and corpus statistics.
  • A shared numeric range does not guarantee comparable relevance meaning.
  • Rank positions are often easier to combine than raw scores.

Fuse two ranked lists with RRF

RRF assigns each document a contribution based on its position in each result list. For a document at rank `r`, the contribution is `1 / (k + r)`. Add contributions across lists, then sort documents by the total. The constant `k` dampens the advantage of a document that appears only at the very top of one list.

A document returned by both regional S3 Vectors dense search and Quickwit BM25 can accumulate evidence from both signals. A document unique to one retriever can still appear in the final results when it ranks highly there. This is useful for queries that mix conceptual language with identifiers, product names, error messages, or quoted terms.

  • Retrieve a bounded top-N list from dense search.
  • Retrieve a bounded top-N list from BM25 search.
  • Use document ID as the key when accumulating RRF scores.
  • Sort by fused score and retain the final candidate count needed by the application.

Make fusion observable and testable

Keep the dense and sparse ranks alongside the fused score in retrieval logs. That record makes it possible to explain why a result was selected and to identify whether a query is primarily semantic, lexical, or supported by both paths.

Evaluate changes with a fixed query set that includes natural-language questions, exact-name lookups, identifiers, and domain-specific phrasing. Review the final ranked results as well as the source lists. RRF reduces dependence on score calibration, but it does not replace relevance evaluation or careful document preparation.

  • Log query ID, document ID, dense rank, BM25 rank, and fused rank.
  • Choose top-N depths large enough to allow overlap and complementary candidates.
  • Treat `k`, dense depth, sparse depth, and final depth as explicit configuration.
  • Test failure cases such as acronyms, version strings, and ambiguous terminology.