Why raw-score blending is fragile

A dense retrieval score represents a relationship in embedding space, while BM25 ranks documents from term frequency and corpus statistics. Even when both systems return numbers that appear comparable, their ranges and distributions can change with the query, index contents, embedding model, and search configuration.

Adding or averaging those scores can therefore make ranking behavior difficult to predict. A query containing an exact identifier may deserve strong BM25 influence, while a conceptual question may be better served by dense retrieval. Rank-based fusion preserves each retriever's ordering without requiring score normalization.

  • BM25 is especially useful for exact terms, codes, names, and rare vocabulary.
  • Dense retrieval can surface semantically related language when wording differs.
  • Raw scores should not be assumed to have equivalent meaning across retrievers.

Retrieve two candidate lists, then apply RRF

For each user query, issue one dense query against the vector index and one sparse query against the BM25 index. Ask each retriever for a candidate set larger than the number of results you plan to display. Then combine documents by their positions in each list.

For a document d, RRF assigns a score of sum(1 / (k + rank_i(d))) over every result list where d appears. The constant k reduces the advantage of a single first-place result and is commonly treated as a tuning parameter. Sort documents by the resulting fused score, using a stable tie-breaker such as document ID or a preferred source rank.

  • Keep document identifiers consistent between the S3 Vectors and Quickwit indexes.
  • Use one-based ranks: the first result has rank 1.
  • Choose a candidate depth that gives both retrievers room to contribute.
  • Deduplicate by canonical document or chunk ID before returning results.

Evaluate failure cases before tuning the fusion constant

Start with a small query set drawn from real user tasks. Include exact-match searches, terminology-heavy questions, paraphrases, ambiguous requests, and queries that should return no useful result. Inspect not only whether a relevant item appears, but whether it is ranked high enough to be seen or passed to a downstream answer-generation step.

If results are weak, diagnose the source before changing RRF. Sparse misses may indicate poor field selection or tokenization. Dense misses may point to chunking, metadata quality, or an embedding mismatch. Fusion tuning is most useful after each individual retriever has a clear and measurable role.

  • Log the dense rank, sparse rank, and fused rank for returned documents.
  • Review queries where only one retriever contributed relevant candidates.
  • Apply metadata filters consistently to both candidate searches when required.
  • Keep a no-result threshold or policy separate from the fusion calculation.