Why raw dense and BM25 scores should stay separate

Dense retrieval ranks documents by proximity between vector representations. BM25 ranks documents using term frequency, document frequency, and length normalization. Both produce ordered results, but a score of 0.8 from one system does not inherently mean the same thing as 0.8 from the other.

Trying to add or average raw scores without calibration can create brittle behavior. A small indexing change, embedding-model change, corpus shift, or BM25 configuration adjustment may alter one score distribution while leaving the other unchanged. The resulting fusion can silently favor one retriever.

For Talqora-backed applications, treat regional S3 Vectors and Quickwit BM25 as independent candidate generators. Their rankings are the stable common interface; their raw score magnitudes are diagnostic data, not a universal relevance currency.

  • Dense search helps with conceptual and paraphrased queries.
  • BM25 helps with identifiers, product names, quoted terms, and rare vocabulary.
  • Score distributions can vary across queries and indexes.
  • Rank position is easier to compare across retrieval methods.

Use reciprocal rank fusion for a reliable first implementation

Reciprocal rank fusion, often shortened to RRF, combines ranked lists without requiring score normalization. For each document, add a contribution from every list in which it appears: 1 divided by k plus its rank. The constant k reduces the gap between very high ranks and lower ranks, preventing a single list from dominating solely because of its first result.

In practice, issue the same user query to dense search and BM25 search, collect a bounded candidate list from each, then merge by document identifier. A document retrieved by both systems receives two contributions. A document retrieved by only one system can still appear, which preserves useful results for specialized queries.

Choose a fixed candidate depth first, such as the number of results your application can reasonably inspect or return, and keep the fusion logic deterministic. This makes relevance investigations much easier: for any result, you can show its dense rank, BM25 rank, and final fused score.

  • For each source list, use ranks starting at 1.
  • Compute: fused_score += 1 / (k + rank).
  • Merge candidates by a stable document or chunk identifier.
  • Sort by fused score, then apply a deterministic tie-breaker.

Make fusion observable before making it more complex

Rank fusion is intentionally simple, but it still needs operational visibility. Record which retriever returned each result, its rank in each list, and the final fused position. These fields let teams distinguish a poor retrieval outcome from a poor fusion outcome.

Build a small evaluation set from real query types: natural-language questions, exact names, acronyms, error messages, part numbers, and mixed queries. Review whether dense retrieval contributes useful semantic alternatives, whether BM25 protects exact matches, and whether the combined top results remain coherent.

Only introduce query-specific weighting or score calibration after you have evidence that a fixed rank fusion policy is insufficient. A transparent baseline is valuable because it gives later tuning work a clear reference point and limits the risk of hiding retrieval regressions behind opaque score transformations.

  • Log source membership and rank for every fused result.
  • Evaluate exact-term and semantic query classes separately.
  • Inspect documents returned by both systems versus only one.
  • Version fusion parameters alongside retrieval configuration.