Why raw-score blending is fragile

Dense retrieval and BM25 produce scores from different models and distributions. A dense similarity score reflects the relationship between embeddings, while a BM25 score reflects term frequency, document frequency, and field statistics. Adding those numbers directly gives one system's scale an accidental advantage.

That advantage can also change as the corpus changes. New documents, revised analyzers, different embedding models, or changes in query wording may alter score ranges even when the underlying relevance behavior remains sensible. A fixed weighting scheme can therefore become difficult to reason about and maintain.

  • Use dense retrieval for semantic similarity and paraphrased queries.
  • Use BM25 for exact vocabulary, codes, names, and rare terms.
  • Avoid treating scores from separate retrieval systems as directly comparable by default.

Apply reciprocal rank fusion to two result lists

RRF combines rankings rather than raw scores. Run the dense query against the S3 Vectors-backed retrieval path and the lexical query against Quickwit BM25, then keep a ranked candidate list from each. For every document, add a contribution based on its position in each list.

A common form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in result list i and k is a positive constant. Documents appearing near the top of either list receive more credit; documents appearing in both lists accumulate credit. If a document is absent from a list, it contributes nothing from that list.

  • Retrieve a bounded top-N list from both dense and sparse search.
  • Deduplicate candidates using a stable document identifier.
  • Compute an RRF score from each candidate's ranks.
  • Sort by fused score and return the required final page.

Choose candidate depth and inspect disagreement

Candidate depth matters because fusion can only promote documents that were retrieved by at least one source. If the dense and sparse lists are too shallow, useful candidates may never enter the fused set. Start with a depth larger than the final number of results, then evaluate using representative queries and relevance judgments where available.

Disagreement between the two lists is diagnostic rather than automatically bad. A sparse-only result may reveal an important exact-token need, while a dense-only result may reveal a synonym or conceptual match. Review these cases to decide whether to improve document text, metadata, query construction, embedding choices, or filtering rules.

  • Log each candidate's dense rank, BM25 rank, and fused rank.
  • Test identifier-heavy, natural-language, and mixed queries separately.
  • Keep filtering and access-control constraints consistent across both retrieval paths.
  • Re-evaluate fusion after corpus, analyzer, or embedding changes.