Why raw dense and BM25 scores should not be added

A dense-search score reflects the similarity measure and embedding model used for vector retrieval. A BM25 score is derived from term frequency, document frequency, field length, and query terms. Even when both systems return a numeric value, the magnitude and distribution of those values describe different ranking functions.

Adding the two scores directly can make one retriever dominate for reasons unrelated to relevance. A safer default is to treat each backend as a ranked candidate generator: regional S3 Vectors provides dense candidates, and Quickwit BM25 provides sparse candidates. Fusion then operates on positions in those lists.

  • Use the same document identifier in dense and sparse indexes.
  • Request a bounded candidate list from each retriever.
  • Preserve each backend's rank order before fusion.
  • Apply filters consistently before results are merged when possible.

Fuse candidate lists with reciprocal rank fusion

For each document d, RRF assigns a combined score by summing 1 divided by k plus the document's rank in every list where it appears: RRF(d) = Σ 1 / (k + rank_i(d)). Documents appearing near the top of either list receive more credit, and documents found by both retrievers receive credit twice.

The constant k reduces the difference between adjacent ranks near the top of a list. A commonly used starting point is 60, but it is a tuning parameter rather than a universal rule. Keep ranks one-based, deduplicate by document ID, sort by the fused score, and return the top results.

  • Run dense search and BM25 search independently for the same query.
  • Record ranks, not just document IDs and raw scores.
  • Add an RRF contribution for every list containing a document.
  • Use a stable secondary sort, such as document ID, when fused scores tie.

Tune the retrieval pipeline with query slices

Evaluate fusion on representative query categories instead of relying on one aggregate impression. Exact-code and identifier queries reveal whether sparse retrieval is retained. Paraphrased, conceptual, and multilingual queries help show where dense retrieval contributes distinct candidates. Queries with filters test whether index metadata and result handling stay aligned.

Start with equal participation from both candidate sets, then inspect failures. If exact matches disappear, increase the BM25 candidate depth or adjust sparse field construction. If semantically relevant documents are missing before fusion, inspect embedding inputs and dense candidate depth. Fusion can only reorder documents that at least one retriever returned.

  • Create a small labeled set covering lexical and semantic query intents.
  • Log dense rank, BM25 rank, fused rank, and final document ID.
  • Review documents unique to each retriever, not only final top results.
  • Change one parameter at a time: candidate depth, k, filters, or field design.