Why raw-score blending is fragile

A dense retriever commonly ranks documents by vector similarity, while BM25 ranks documents using term-frequency and document-statistics signals. Even when both produce a numeric score, the magnitude and distribution of those scores need not mean the same thing.

Adding or averaging raw scores can therefore make one retriever dominate for accidental reasons, such as a wider score range. It can also make relevance behavior change when an index is rebuilt, a similarity metric changes, or corpus statistics shift.

  • Dense retrieval can surface semantically related wording.
  • BM25 can reward exact identifiers, rare terms, and quoted phrases.
  • Score scales should not be assumed comparable across retrieval methods.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its rank in each result list. For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank from retrieval list i and k is a positive constant.

Run dense retrieval against the S3 Vectors-backed index and BM25 retrieval against Quickwit, request a candidate list from each, then merge document identifiers and sum their RRF contributions. A document appearing in both lists receives two contributions; a document that ranks highly in one list can still remain competitive.

  • Use one-based ranks: first place has rank 1.
  • Choose the same candidate depth for both lists as a simple starting point.
  • Use a fixed k initially, then evaluate it with representative queries.
  • Deduplicate by a stable document or chunk identifier before returning results.

Make fusion observable and testable

Log the dense rank, BM25 rank, and final fused rank for returned documents. These fields make it possible to distinguish a result supported by both retrievers from one carried by only dense or sparse retrieval, which is useful when investigating misses.

Evaluate fusion on a small, curated query set before changing defaults. Include semantic paraphrases, product names, error codes, acronyms, and multi-term questions. The goal is not to prove that one retriever always wins, but to verify that the merged ranking handles the query types your users actually submit.

  • Inspect queries where exact terms matter, such as IDs and configuration keys.
  • Inspect paraphrased queries where vocabulary differs from the source text.
  • Keep retrieval depth, k, and evaluation judgments versioned.
  • Re-test after corpus, chunking, embedding, or analyzer changes.