Treat dense and sparse scores as separate signals

A cosine-like dense similarity score and a BM25 score are produced by different retrieval models and have different distributions. A larger number in one system does not necessarily mean the same thing as a larger number in the other. Adding those raw values together can make ranking behavior depend more on score scale than on relevance.

Rank positions are easier to compare. A document placed first by either retriever has received a strong vote; a document placed far down the list has received a weaker one. RRF works with those positions rather than requiring a global score normalization strategy.

  • Use dense retrieval for semantic similarity and paraphrased queries.
  • Use BM25 for exact names, codes, quoted terms, and uncommon vocabulary.
  • Keep the source-specific score and rank in retrieval logs for later debugging.

Fuse the result lists with a small, explicit formula

For each query, retrieve a bounded candidate list from S3 Vectors and another from Quickwit BM25. Assign every document a fused score by summing 1 divided by k plus its rank for each list in which it appears. In notation: RRF(d) = Σ 1 / (k + rank_i(d)).

The constant k reduces the difference between adjacent high ranks and lower ranks. A common starting value is 60, but it is a tuning choice rather than a universal rule. Select it using representative queries and relevance judgments, then keep it stable long enough to observe its behavior.

  • Use one-based ranks: first place has rank 1.
  • Deduplicate by a stable document or chunk identifier before returning results.
  • Choose a candidate depth that gives both retrievers a meaningful chance to contribute.
  • Return the fused order while retaining which retriever contributed each result.

Make fusion observable before making it complex

Start with RRF before introducing learned rankers or query-dependent weighting. It offers a clear explanation for each result: the item ranked well in dense retrieval, sparse retrieval, or both. That clarity is useful when a result looks surprising or when content changes alter retrieval behavior.

Evaluate query groups separately. A documentation query containing an error code may benefit disproportionately from BM25, while a natural-language support question may receive more useful candidates from dense search. The goal is not to make both lists look identical; it is to improve the final candidate set for the next stage of the application.

  • Log query text, candidate identifiers, source ranks, and fused rank.
  • Review failures by intent: exact lookup, semantic exploration, and mixed queries.
  • Watch for duplicate chunks from the same source crowding the final list.
  • Add reranking only after you can explain the quality of the fused candidate set.