Why raw scores should not be compared directly

Dense retrieval commonly ranks documents by a vector-similarity measure, while BM25 ranks them using term statistics and document-length normalization. Even when both methods return a numeric value, those values arise from different calculations and do not naturally share a meaningful scale.

Adding or averaging uncalibrated dense and BM25 scores can therefore make ranking sensitive to score ranges rather than relevance. A safer initial design is to preserve each retriever’s ordered results and combine rank positions or independently normalized signals only after measuring the effect on representative queries.

  • Dense search can surface semantically related wording that does not share exact terms with the query.
  • BM25 can strongly reward precise identifiers, names, error codes, and uncommon query terms.
  • A score of 0.8 from one retriever is not inherently equivalent to 0.8 from another.
  • Keep source-specific scores and ranks in retrieval logs for later analysis.

Generate candidates in parallel, then fuse deliberately

Run dense and sparse retrieval independently against the same document corpus, requesting a bounded candidate list from each. The union of those lists becomes the set available for downstream ranking; documents returned by both retrievers gain useful evidence from two different signals.

A rank-based fusion method is a practical baseline because it does not require the dense and BM25 scores to be numerically comparable. For example, reciprocal-rank fusion assigns each document a contribution based on its position in each result list, then sums contributions for documents that appear more than once.

  • Choose separate candidate depths for dense and sparse retrieval rather than assuming one depth fits both.
  • Deduplicate candidates by a stable document or chunk identifier before ranking.
  • Record which retrieval paths contributed each candidate.
  • Use a deterministic tie-breaker, such as document ID, to make results reproducible.

Tune with query classes instead of one global intuition

Hybrid retrieval is most useful when query types differ. A natural-language question may benefit from semantic recall, while a search for a configuration key or product code may depend on lexical precision. Build a small evaluation set that explicitly includes both cases, along with ambiguous and multi-term queries.

Review failures at the candidate stage before changing fusion logic. If a relevant item never enters either candidate list, ranking cannot recover it. If it appears in both lists but ranks poorly, then candidate depth, fusion weights, chunking, metadata filters, or a later reranking stage are clearer areas to investigate.

  • Label queries by intent, such as conceptual question, exact lookup, troubleshooting, or navigation.
  • Measure whether a known relevant result enters the fused candidate set before measuring final rank.
  • Inspect dense-only, BM25-only, and overlap results for each failed query.
  • Version evaluation queries and relevance judgments alongside retrieval configuration changes.