Why raw dense and sparse scores should not be added directly

A dense retrieval score and a BM25 score are produced by different models with different scales, distributions, and meanings. A cosine-like similarity may occupy a narrow numerical range, while BM25 values can vary substantially with query length, term rarity, and document length. Adding those values together creates a hidden weighting decision that can change from one query to the next.

Score normalization can be useful, but it requires careful calibration and ongoing monitoring. Before introducing that complexity, preserve what each retriever reliably provides: an ordering of likely results. Rank fusion uses those orderings and avoids assuming that a score of 0.8 from one retrieval method is comparable to a score of 8 from another.

  • Dense retrieval helps with paraphrases and concept-level matches.
  • BM25 helps with exact terms such as product names, IDs, acronyms, and error messages.
  • Raw score ranges are not inherently comparable across retrieval methods.
  • Rank-based fusion is a practical baseline when calibrated relevance labels are limited.

Apply reciprocal rank fusion to two candidate lists

Run the same user query through the dense and sparse retrieval paths, then collect a fixed number of candidates from each. Match candidates using a stable document or chunk ID. For every result, add a contribution based on its position in each list: RRF(d) = Σ 1 / (k + rank_i(d)). The sum is taken over the retrieval lists in which document d appears.

The constant k softens the advantage of the first few positions. A commonly used starting value is 60, but it is a tuning parameter rather than a universal rule. After calculating fused scores, sort descending and return the top results. A document appearing reasonably high in both lists will often outrank one that appears only in one list, while a strong exact-match or semantic-only result can still survive.

  • Retrieve a candidate set from S3 Vectors and a candidate set from Quickwit BM25.
  • Use one canonical ID for each retrievable chunk across both indexes.
  • Calculate 1 / (k + rank) for each list where a candidate appears.
  • Sum contributions, sort by the fused value, and pass the top results to the next stage.

Choose candidate depth and inspect disagreement

Candidate depth matters because fusion cannot promote a document that neither retriever returns. Start with a depth larger than the final number of results you display—for example, retrieve several times more candidates than the final top-k—then evaluate whether relevant documents are being missed before fusion. The appropriate depth depends on corpus size, query type, and latency constraints.

Do not treat fusion as a one-time configuration. Log the dense rank, sparse rank, fused rank, and canonical chunk ID for sampled queries. Queries where the retrieval methods disagree are especially valuable: they reveal whether users depend on exact terminology, whether chunk text lacks context, or whether embeddings are not representing an important domain distinction.

  • Keep chunk boundaries and canonical IDs consistent between dense and sparse indexes.
  • Deduplicate candidates before presenting results.
  • Review queries with large dense-versus-sparse rank differences.
  • Tune k and candidate depth against judged queries or carefully reviewed search samples.