Start with the failure modes, not the retrieval labels

Dense retrieval is valuable when a query and a document use different words to express a related idea. An embedding can bring together concepts such as “account access issue” and “cannot sign in,” even when their wording does not overlap exactly.

Sparse retrieval is valuable when wording is itself meaningful. Product identifiers, error codes, names, quoted phrases, uncommon terminology, and recently introduced vocabulary can all be strong BM25 signals. A query containing an exact token should not depend entirely on an embedding representation preserving that token’s importance.

The engineering goal is therefore not to decide whether dense or sparse retrieval is universally better. It is to identify which classes of query each method can rescue when the other method produces weak candidates.

  • Use dense retrieval for conceptual similarity and paraphrases.
  • Use BM25 for exact terms, rare tokens, and lexical constraints.
  • Inspect queries that return plausible but incorrect results.
  • Treat search quality as a query-distribution problem.

Generate candidates independently before combining them

A practical hybrid design begins by asking both retrievers for candidates. Dense retrieval searches vector representations, while BM25 ranks documents according to term-based relevance. Keeping the first retrieval stage independent preserves the strengths of both signals.

The two result sets can then be merged by document identity. Documents found by both methods are often useful signals: they have semantic and lexical support. Documents found by only one method should not automatically be discarded, because a one-sided match may be exactly what a specialized query needs.

This approach also makes diagnosis easier. If a desired document is absent from both candidate sets, the issue may be document ingestion, chunking, filtering, or query formulation. If it appears in one set but ranks poorly after merging, the issue is more likely in the combination rule.

  • Retrieve a candidate set from dense search and a candidate set from BM25.
  • Deduplicate candidates using a stable document or chunk identifier.
  • Record which retrieval path contributed each candidate.
  • Apply the same required metadata filters consistently across retrieval paths.

Prefer rank-aware fusion when score scales differ

Dense similarity values and BM25 scores are not inherently on the same scale. Directly adding raw scores can produce unstable behavior because a change in embedding model, corpus composition, or BM25 configuration may alter score distributions without changing the underlying relevance judgment.

A rank-aware fusion rule avoids assuming that the numerical scores are comparable. Instead, it rewards documents according to their position in each result list. The exact fusion formula is a product decision, but the important property is that it combines relative ranking evidence rather than treating unrelated score scales as interchangeable.

Evaluate the combined ranking with a query set that includes natural-language questions, identifier-heavy queries, short queries, and ambiguous queries. The target is not merely higher aggregate relevance; it is fewer cases where an exact match is missed or a semantically relevant result is buried.

  • Avoid assuming BM25 and dense scores can be added without calibration.
  • Use ranks or carefully normalized scores for fusion.
  • Compare dense-only, sparse-only, and hybrid result sets during evaluation.
  • Review failures by query type before changing weights or candidate limits.