Why raw dense and sparse scores should not be added blindly

A dense-search score reflects the relationship between an embedding query and an embedding vector. A BM25 score is based on term statistics and document length. Even when both outputs are useful, their numeric ranges and distributions are not inherently aligned.

Adding the two scores with a fixed weight can make one retriever dominate for reasons unrelated to relevance. A weight that appears reasonable for one corpus, query type, or embedding model may behave differently after content changes or an indexing configuration is adjusted.

  • Dense retrieval is often useful for paraphrases and conceptually related language.
  • BM25 is often useful for exact names, error codes, product terms, and uncommon tokens.
  • Score magnitude alone is not a universal measure of relevance across retrieval methods.

Fuse ranks, not scores, with RRF

Run the same user query through both retrieval paths: dense search against S3 Vectors and sparse search through Quickwit BM25. Keep a ranked list from each path, then assign each document an RRF contribution based on its position in each list.

For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) for every list in which d appears. The constant k reduces the difference between nearby positions near the top of a list. Documents returned by both retrievers accumulate contributions, while a strong result from only one retriever can still remain competitive.

  • Use one-based ranks: first result has rank 1.
  • Deduplicate by a stable document or chunk identifier before returning results.
  • Choose a candidate depth for each retriever before fusion, such as the top N results.
  • Keep the dense and BM25 result lists available for debugging.

Make hybrid behavior observable and tune it with query sets

RRF removes the need to normalize raw scores, but it does not remove the need for evaluation. Build a small query set that represents real retrieval needs: natural-language questions, exact identifiers, short keyword queries, and mixed queries containing both concepts and literal terms.

For each query, record which source retrieved the selected result, its rank in each source list, and its final fused rank. This makes failures actionable. If exact identifiers are routinely missing, inspect tokenization and sparse indexing. If paraphrased questions miss relevant material, inspect chunking, embedding generation, and the dense candidate depth.

  • Log per-source rank and final fused rank for returned documents.
  • Review queries where only one retrieval path contributes useful candidates.
  • Test candidate depth changes separately from changes to the RRF constant.
  • Use stable relevance judgments when comparing retrieval configuration changes.