Retrieve from each signal before combining results

A common mistake in hybrid search is to treat dense and sparse scores as though they are directly comparable. They usually are not. Each retrieval method has its own scoring behavior, corpus sensitivity, and query-dependent distribution, so a raw score from one system should not automatically outweigh a raw score from the other.

Instead, issue a dense query and a BM25 query independently. Request a candidate set from each path, preserve the rank and source of every returned document, and merge by a stable document identifier. This produces a combined candidate pool while retaining evidence about why each document appeared.

  • Send the embedding query to the dense retrieval path.
  • Send the original or lightly normalized text query to BM25.
  • Fetch more candidates than the final number of results needed.
  • Store document ID, source path, rank, and any available score for debugging.

Use rank-based fusion as a safe first baseline

Rank-based fusion avoids assuming that dense and BM25 scores share a common scale. One practical approach is reciprocal rank fusion (RRF): for each document, add a contribution based on its rank in each result list. Documents that rank well in either list, or reasonably well in both, rise in the fused ordering.

A typical form is RRF(d) = Σ 1 / (k + rank_i(d)), where the sum is over retrieval lists containing document d. The constant k softens the impact of the first few ranks. Its exact value is a tuning choice, but the important property is that the formula uses positions rather than raw scores.

  • Deduplicate documents by a canonical ID before final ranking.
  • Assign no contribution for a document absent from a retrieval list.
  • Keep separate dense and sparse ranks in logs or response metadata.
  • Start with equal signal weighting; change weights only after reviewing representative queries.

Evaluate failures by query type, not only aggregate relevance

Hybrid retrieval is most useful when it improves distinct query classes. Exact product names, codes, quoted fragments, and uncommon terminology often benefit from sparse matching. Broad questions, paraphrases, and concept-oriented searches may benefit from dense retrieval. A single aggregate metric can hide regressions in one of these groups.

Build a small evaluation set that labels query intent and includes known relevant documents. Review the top results from dense-only, sparse-only, and fused retrieval side by side. When a fused result is surprising, inspect whether the issue came from candidate generation, document chunking, metadata filtering, or fusion—not just the final ordering.

  • Include exact-match, semantic, acronym, and multi-topic queries.
  • Record whether a relevant item was missing from candidates or merely ranked too low.
  • Test metadata filters with both retrieval paths when filters are part of the request.
  • Version fusion settings so relevance changes can be reproduced.