Why raw dense and BM25 scores should not be added blindly

A dense-search score and a BM25 score are produced by different retrieval models with different scales and distributions. A score of 0.8 from one dense-search configuration does not have an inherent relationship to a BM25 score of 8.0. That relationship can also change when embeddings, analyzers, fields, or corpus composition change.

Talqora uses regional S3 Vectors for dense search and Quickwit BM25 for sparse search. Treat the outputs as two ranked candidate lists. This keeps the retrieval layer focused on what each method does well instead of relying on an assumed cross-system score conversion.

  • Dense search can help with paraphrases and concept-level similarity.
  • BM25 can preserve lexical precision for names, error codes, and uncommon terms.
  • Score scales may change after index or query-processing changes.
  • Rank positions are easier to combine consistently than raw scores.

Apply Reciprocal Rank Fusion to the two result lists

RRF assigns each document a fusion contribution based on its position in each list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of d in result list i. A document returned by both searches receives contributions from both lists.

Use the same document identifier in the dense and BM25 indexes so duplicate detection is deterministic. Fetch a bounded candidate set from each retrieval path, combine candidates by identifier, sum their RRF contributions, then sort descending by the fused value. The constant k reduces the influence of small differences near the top of a list; choose it deliberately and keep it configurable.

  • Run dense and BM25 retrieval for the same query.
  • Request a candidate window from each path, such as the top N results.
  • Deduplicate by stable document or chunk ID.
  • Sum each candidate's rank-based RRF contributions and return the highest fused results.

Validate with a query set before making fusion the default

Build a small evaluation set from representative searches, including exact identifiers, short ambiguous queries, natural-language questions, and queries containing domain terminology. For each query, record whether the relevant item appears in the final top positions for dense-only, BM25-only, and fused retrieval.

Inspect failures rather than relying only on aggregate measures. If BM25 results are often dominated by boilerplate, improve field selection or text preparation. If dense results repeatedly retrieve adjacent but wrong concepts, review chunk boundaries and embedding inputs. RRF is a robust merging step, but it cannot repair irrelevant candidate lists.

Keep the dense and sparse candidate sizes observable in application logs. When a query produces candidates from only one path, that is useful diagnostic information: it may indicate a vocabulary mismatch, missing indexed content, or a query that is inherently better served by one retrieval method.

  • Evaluate dense-only, BM25-only, and fused rankings on the same queries.
  • Include lexical queries such as product names, codes, and quoted phrases.
  • Record which retrieval path contributed each final result.
  • Revisit candidate-window sizes and k when corpus or query patterns change.