Why raw dense and BM25 scores should not be added

A dense search score expresses proximity in an embedding space, while BM25 is based on term frequency, document frequency, and field-length normalization. Even when both systems return a numeric score, the values do not automatically have the same meaning or range.

Adding those scores directly can make ranking sensitive to implementation details such as embedding similarity choice, BM25 configuration, query length, and candidate-set composition. A query containing an exact identifier may need strong lexical matching, while a paraphrased question may benefit more from dense retrieval.

  • Dense retrieval can surface semantically related wording.
  • BM25 can preserve exact terms, identifiers, error codes, and uncommon names.
  • Raw-score weighting requires calibration and ongoing validation.
  • Rank-based fusion avoids assuming score comparability.

Fuse ranked lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus the rank of d for every list in which it appears. The constant k reduces the difference between nearby ranks and keeps a single first-place result from dominating too aggressively.

Fetch a bounded candidate list from regional S3 Vectors and another from Quickwit BM25, then deduplicate by a stable document or chunk identifier. Sum each candidate's rank contributions, sort by the fused score, and return the top results or pass them to a later reranking stage if one is available in your application.

  • Choose the same initial candidate depth for both retrievers as a clear starting point.
  • Use rank positions starting at 1, not 0.
  • Only add a contribution when a document appears in that retriever's list.
  • Keep the dense and BM25 ranks in retrieval logs for debugging.

Make fusion observable and tune it with representative queries

RRF is deliberately simple, but it is still a retrieval policy. Evaluate it with a query set that includes natural-language questions, exact-title searches, product or entity names, abbreviations, and typo-prone inputs. For each query, inspect whether relevant documents appear in the dense list, the BM25 list, both, or neither.

When results are weak, diagnose the retrieval stage before changing fusion. Missing dense candidates may point to chunking, embedding, or metadata issues. Missing lexical candidates may point to tokenization, field selection, or indexing choices. Fusion can promote candidates that exist; it cannot recover documents absent from both lists.

  • Record per-source rank and final fused rank for returned documents.
  • Review queries where dense-only and BM25-only results disagree.
  • Test candidate depth and the RRF k value against a fixed evaluation set.
  • Treat metadata filters consistently across both retrieval paths.