Why raw-score mixing is fragile

Dense retrieval scores are produced from vector similarity, while BM25 scores are derived from term-frequency and document-length statistics. Even when both lists are useful, their numeric ranges and distributions do not necessarily mean the same thing.

Adding or averaging raw scores can therefore make one retriever dominate for accidental scaling reasons rather than because it found more relevant documents. This is especially risky when an index, embedding model, query type, or scoring configuration changes.

Talqora uses regional S3 Vectors for dense search and Quickwit BM25 for sparse search. Treating their outputs as ranked lists gives an application a stable integration point across these distinct retrieval methods.

  • Dense search can surface semantically related wording.
  • BM25 can strongly reward exact terms, identifiers, and rare phrases.
  • Score magnitudes should not be assumed to be directly comparable.

Fuse candidates with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in each result list. For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is its one-based rank in retriever i and k is a positive constant chosen by the application.

Retrieve a bounded candidate list from dense search and another from BM25, deduplicate documents by a stable document ID, sum their RRF contributions, and sort descending. A document appearing in both lists receives credit from both signals; a document highly ranked by just one list can still compete.

The constant k reduces the gap between adjacent ranks. Rather than treating it as a universal setting, keep it explicit in configuration and evaluate it against representative queries and relevance judgments.

  • Use one-based ranks: the first result has rank 1.
  • Deduplicate before presenting results, while retaining contributions from both lists.
  • Apply a deterministic tie-breaker, such as document ID, for repeatable ordering.
  • Keep the source rank and fused score in logs for debugging.

Make fusion observable and testable

Hybrid retrieval is easier to operate when each result carries retrieval metadata: whether it came from dense search, BM25, or both; its rank in each list; and its final fused score. This makes surprising results inspectable without trying to infer causes from a single opaque number.

Create a small evaluation set that includes semantic paraphrases, exact-name lookups, version strings, error messages, and short ambiguous queries. These query families reveal different failure modes: dense retrieval may miss exact tokens, while lexical retrieval may miss alternate wording.

If the fused list is later passed to a reranker or generation system, preserve the retrieval-stage evidence separately. Candidate selection and downstream ranking are different decisions, and separating them makes regressions easier to localize.

  • Log candidate counts and overlap between dense and BM25 lists.
  • Review queries where only one retriever supplied the top result.
  • Version embedding models, BM25 configuration, and fusion settings independently.
  • Re-run a fixed query set after changes to ingestion or retrieval logic.