Why dense and sparse scores do not naturally mix

Dense retrieval ranks documents by the relationship between embedding vectors. BM25 ranks documents from lexical evidence such as matching terms, term frequency, and document statistics. Even when both systems return a numeric score, those numbers arise from different models and scales.

Adding or averaging uncalibrated scores can produce unstable rankings. A small change in an embedding model, corpus composition, BM25 configuration, or query wording may alter score distributions without indicating that one retrieval signal has become more important.

  • Dense search can surface semantically related language that does not share exact terms.
  • BM25 can strongly reward precise identifiers, error messages, names, and rare terminology.
  • A score of 0.8 from one retriever has no inherent equivalence to 0.8 from another.
  • Rank positions are usually easier to combine safely than raw scores.

Fuse result lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in every result list where it appears. For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs over the dense and BM25 lists, rank_i is the one-based rank in a list, and k is a positive constant that reduces the impact of very early rank differences.

For example, request a candidate list from regional S3 Vectors and another from Quickwit BM25. Deduplicate by a stable document or chunk ID, calculate the RRF score for each candidate, then sort descending. A document that ranks well in both lists is promoted, while a document that is highly ranked by only one method can still remain competitive.

  • Use the same canonical chunk ID in dense and sparse indexes.
  • Retrieve more candidates than the final number displayed or passed downstream.
  • Choose one k value and evaluate it on representative queries before changing it.
  • Preserve source ranks in logs so unexpected fused results can be inspected.

Make fusion observable and test it by query type

Hybrid retrieval is most useful when evaluation reflects the language people actually use. Build a small query set that includes exact identifiers, natural-language questions, acronym-heavy requests, and queries whose relevant wording differs from the source text. Record which chunks should be retrieved, not only whether an answer sounds plausible.

Inspect the dense-only, BM25-only, and fused top results side by side. This makes failures actionable: missing exact matches may point to sparse indexing or analysis choices, while semantic misses may point to chunking, embedding selection, or document preparation. RRF itself is deliberately simple, so it provides a clear baseline before adding more ranking complexity.

  • Track whether a relevant chunk appears in either candidate list before fusion.
  • Measure top-k retrieval outcomes separately for exact-match and semantic query groups.
  • Log document IDs, ranks, and fusion contributions for each returned result.
  • Re-run the same evaluation set after corpus, chunking, or retrieval configuration changes.