Why raw-score blending is usually a fragile starting point

A dense retriever and BM25 produce scores with different meanings and distributions. A vector similarity score reflects the relationship between embeddings, while BM25 is driven by term frequency, document frequency, and document length. Adding those values together assumes they share a stable scale, which is often not true across queries or corpus changes.

Score normalization can help, but it introduces more assumptions to validate. A simpler initial design is to preserve the order from each retriever and combine ranks instead. This lets the system benefit from both signals without interpreting a BM25 value as if it were directly equivalent to a vector similarity value.

  • Dense retrieval helps when a query and a relevant document use different language.
  • BM25 helps when exact terms, model numbers, names, and rare tokens are decisive.
  • Rank-based fusion avoids relying on a shared score scale.
  • Independent result lists make retrieval behavior easier to inspect.

Use reciprocal rank fusion for a dependable first hybrid pass

Reciprocal rank fusion, commonly abbreviated RRF, assigns each document a contribution based on its position in each result list. For a document d, the fused score can be written as: RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs across retrievers, rank_i(d) is the document position in a given list, and k is a positive constant that reduces the influence of very high ranks.

In practice, retrieve a bounded top-N list from regional S3 Vectors and a bounded top-N list from Quickwit BM25. Join results by a stable document or chunk identifier, calculate an RRF score for every candidate found in either list, then sort by that score. A document that appears near the top in both lists will naturally rise, while a strong result from only one method can still be retained.

  • Choose one canonical identifier for joining dense and sparse results.
  • Keep the initial retrieval depth explicit, such as top-N per retriever.
  • Treat missing documents as having no contribution from that retriever.
  • Record per-retriever rank alongside the fused rank for debugging.

Evaluate disagreement, not just aggregate relevance

Hybrid retrieval is most valuable on queries where the two methods disagree. Build a small evaluation set that includes exact-title searches, acronyms, identifiers, paraphrases, short natural-language questions, and domain-specific terminology. For each query, compare dense-only, BM25-only, and fused result lists against the same relevance judgments.

The goal is not to prove that one retriever always wins. Instead, identify the query classes that benefit from fusion and the cases where it introduces noise. Inspecting retrieved chunks is essential: a result can contain the right words yet lack the answer, or be semantically related while omitting a required constraint. Those observations guide later work on chunking, metadata filtering, query routing, or reranking.

  • Review queries where only dense retrieval found the relevant item.
  • Review queries where only BM25 found the relevant item.
  • Track whether fusion preserves important exact-match results.
  • Version evaluation queries and relevance labels as the corpus evolves.