Why raw dense and BM25 scores should not be added casually

A dense retrieval score reflects the similarity function and embedding representation used by the vector index. A BM25 score reflects term frequency, document frequency, field length, and query terms. Even when both are useful relevance signals, a score of 0.8 from one system does not inherently mean the same thing as 0.8 from the other.

Adding raw scores can make ranking sensitive to implementation details rather than relevance. A change to an embedding model, a BM25 configuration, or a search field can alter score distributions and unexpectedly shift hybrid results. Score normalization is possible, but it requires careful evaluation and ongoing maintenance.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful for exact vocabulary, product names, error codes, and rare terms.
  • Each retrieval method should return an ordered candidate list.
  • Treat scores as method-specific unless they have been deliberately calibrated.

Fuse rankings with Reciprocal Rank Fusion

RRF combines lists by rewarding documents that appear near the top of one or more rankings. For each document, add a contribution based on its rank in every list where it appears: 1 divided by k plus the rank. The constant k reduces the difference between nearby ranks and prevents the first position from dominating too aggressively.

In a Talqora retrieval flow, a service can issue the same user query to dense search backed by regional S3 Vectors and sparse search backed by Quickwit BM25. It can then deduplicate document identifiers, calculate an RRF score for the returned ranks, and sort the merged candidates before passing a small set to an application or downstream reranker.

  • Use a stable document ID to match the same item across dense and sparse results.
  • Choose a candidate depth for each retriever before fusion, such as the top N results.
  • Calculate: RRF(document) = Σ 1 / (k + rank).
  • Start with one shared k value, then evaluate changes using representative queries.

Build an evaluation loop around query intent

Hybrid retrieval is most valuable when queries vary. A support corpus may contain natural-language questions, exact error messages, API field names, and versioned identifiers. Build an evaluation set that includes each of these patterns rather than judging fusion only on broad semantic questions.

Inspect failures by retrieval source. If BM25 finds a critical exact-match document that dense search misses, fusion should preserve it. If dense search retrieves a useful paraphrase absent from lexical results, fusion should also give it a path into the final set. This analysis helps determine candidate depth, k, and whether a later reranking stage is necessary.

  • Record the dense rank, sparse rank, fused rank, and final selected documents for test queries.
  • Include short queries, long questions, identifiers, quoted phrases, and misspellings where relevant.
  • Evaluate recall in the candidate set before optimizing presentation order.
  • Re-run the evaluation when changing embeddings, indexed content, BM25 fields, or fusion parameters.