Why dense and sparse scores should not be added blindly

A dense search result is ranked by a vector similarity measure, while BM25 ranks documents from term-frequency and corpus statistics. Both produce ordered results, but the numerical values have different meanings and may change as indexes, embeddings, or query distributions change.

Adding a dense score to a BM25 score can therefore make retrieval sensitive to arbitrary scaling choices. A fixed weight that looks reasonable for one collection or query type may overemphasize one retriever for another.

  • Dense retrieval can recover semantic matches with little word overlap.
  • BM25 can strongly surface exact names, error codes, model numbers, and quoted text.
  • Score ranges alone do not establish that two ranking signals are calibrated.

Use reciprocal rank fusion as a stable first merge

Reciprocal rank fusion, or RRF, combines result lists using positions rather than raw scores. For each document returned by a retriever, add 1 divided by a constant plus that document's rank. Documents appearing near the top of either list receive a useful boost, and documents found by both retrievers accumulate evidence.

For a query, retrieve a candidate list from dense search in regional S3 Vectors and another from Quickwit BM25. Keep a document identifier shared by both indexes, then merge the ranked lists in the application layer. A common form is RRF(d) = sum of 1 / (k + rank_i(d)), where k dampens the difference between adjacent ranks.

  • Use one canonical document or chunk ID across the dense and sparse indexes.
  • Choose candidate depths before fusion, such as retrieving the top N from each retriever.
  • Deduplicate by canonical ID before returning final results.
  • Treat the RRF constant and candidate depth as relevance-tuning parameters, not universal defaults.

Make fusion observable before making it complex

Log which retriever contributed each final result, its original rank, and its fused rank. These fields make it possible to inspect whether exact-term queries are being carried by BM25, whether semantic queries are being carried by dense retrieval, and whether one branch is rarely contributing useful candidates.

Evaluate with a small set of representative queries and judged relevant results. Include queries with product names, acronyms, natural-language questions, misspellings, and domain-specific terms. If failures cluster around a query class, adjust chunking, metadata filtering, candidate depth, or query handling before introducing more elaborate score normalization.

  • Track dense-only, BM25-only, and overlapping candidates in the final top results.
  • Review queries where a relevant item appears in one candidate list but disappears after fusion.
  • Version embeddings, chunking rules, and index-generation settings alongside evaluation results.
  • Apply metadata filters consistently to both retrieval branches when the filter defines the searchable corpus.