Why raw hybrid scores are difficult to combine

Dense retrieval represents the relationship between a query and a document through vector similarity or distance. BM25, by contrast, ranks documents from term occurrence, term frequency, document frequency, and document-length effects. Even when both searches return numeric scores, the numbers are not automatically comparable.

A simple weighted sum can therefore be fragile. A change to embedding generation, vector similarity configuration, text analysis, corpus composition, or BM25 settings may alter score distributions. The application then has to revisit weights and thresholds, even if the quality of each individual retriever has not materially changed.

  • Dense retrieval is useful for semantic similarity and paraphrases.
  • BM25 is useful when exact terms, identifiers, names, and rare tokens matter.
  • A score of 0.8 from one retrieval method does not inherently mean more than a score of 8 from another.
  • Rank position is often a more stable signal for first-pass fusion than raw score magnitude.

Fuse ranked lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across the lists where d appears. The constant k reduces the impact of small rank differences near the top of a list.

For example, an application can issue a dense query against regional S3 Vectors and a sparse BM25 query against Quickwit, request a candidate list from each, deduplicate by a stable document or chunk ID, and sum each candidate’s RRF contributions. Sort the combined candidates by that total before returning them or passing them to a later reranking stage.

  • Use one-based ranks: the first result has rank 1.
  • Choose a single k value and keep it fixed while evaluating changes.
  • Give a missing document no contribution from that result list.
  • Deduplicate before presenting results, using the same identifier across dense and sparse indexes.

Make fusion an observable application-layer step

Treat fusion as a small, explicit part of the retrieval pipeline. Log the query, the dense rank, the BM25 rank, the fused rank, and the document identifier for sampled requests. This makes it possible to investigate why a result surfaced and to spot cases where one retriever consistently dominates the final list.

Evaluate with queries that reflect real retrieval needs, especially acronym-heavy questions, product or policy names, error messages, and paraphrased natural-language requests. RRF does not remove the need for evaluation, but it avoids assuming that independently produced score scales can be safely added together.

  • Keep dense and sparse candidate depths large enough to create useful overlap and complementary coverage.
  • Inspect zero-overlap queries; they can reveal vocabulary gaps, chunking problems, or embedding mismatch.
  • Record which source lists contributed to each fused result.
  • Use relevance judgments or review samples to compare fusion settings before changing production behavior.