Why raw-score merging is a fragile default

A dense retrieval score reflects the relationship between a query embedding and a document embedding. A BM25 score is derived from term matches, term rarity, and document-length normalization. Even when both systems return numeric scores, the numbers do not necessarily have the same range, distribution, or meaning.

Sorting one combined list by raw score can therefore produce accidental behavior. A small change to tokenization, embedding model, corpus composition, or retrieval configuration may shift one score distribution enough to dominate the other. This can happen even when the underlying relevance of each retriever has not meaningfully changed.

  • Use dense search for semantic similarity and paraphrased language.
  • Use BM25 for exact terminology, product names, codes, and rare phrases.
  • Avoid assuming a BM25 score of one value is equivalent to a dense similarity score of another value.

Build a candidate union, then apply reciprocal rank fusion

Retrieve a ranked candidate list from each retrieval path. In Talqora’s architecture, dense candidates are backed by regional S3 Vectors and sparse candidates by Quickwit BM25. Preserve each list’s rank position, deduplicate documents by a stable document identifier, and calculate a fusion score from rank rather than from the original retrieval score.

Reciprocal rank fusion, commonly abbreviated RRF, adds a contribution for every list in which a document appears. A typical formula is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in retrieval list i and k is a positive constant. The constant reduces the difference between adjacent top ranks and keeps the method from overreacting to a small rank movement.

  • Fetch the same candidate depth from dense and sparse retrieval as an initial baseline.
  • Treat missing documents as contributing zero from that retrieval list.
  • Deduplicate before returning results so a document appears once with its combined fusion score.
  • Keep the original dense and BM25 ranks in logs for later debugging.

Tune the pipeline with query slices, not a single average

RRF has few moving parts, but it still benefits from deliberate evaluation. Create a small set of representative queries and label them by retrieval pattern: exact identifiers, short ambiguous queries, natural-language questions, multilingual phrasing, and queries containing both product terms and descriptive intent. Compare dense-only, BM25-only, and fused rankings on each slice.

Start with equal treatment for both retrieval lists, then adjust only when evidence supports it. If exact-code queries consistently require sparse evidence, increase the sparse candidate depth or introduce a second sparse contribution. If semantic question answering is the main workload, ensure the dense candidate set is deep enough to surface relevant paraphrases. Make one change at a time and retain query-level examples alongside aggregate relevance judgments.

  • Inspect failures where a relevant result appears in only one retrieval list.
  • Track candidate depth, rank positions, document IDs, and the final fusion order.
  • Re-evaluate after changing embeddings, analyzers, document chunking, or corpus content.
  • Use rank fusion as a transparent baseline before adding more complex reranking.