Why raw-score blending is fragile

A dense-search score and a BM25 score are not automatically comparable. Their scale, distribution, and sensitivity to query length can differ, even before accounting for index configuration or embedding-model changes. Adding the two values together can make one retrieval path dominate for reasons unrelated to relevance.

Normalizing scores can help in tightly controlled systems, but it introduces assumptions that need ongoing validation. A more dependable starting point is to treat each retriever as a source of ranked candidates rather than as a source of directly comparable numeric evidence.

This is particularly useful for mixed corpora. A natural-language question may benefit from semantic matches in dense retrieval, while a search containing a product code, error message, person name, or quoted phrase may depend on sparse lexical matching.

  • Dense retrieval ranks documents by vector similarity.
  • BM25 ranks documents using term-frequency and corpus statistics.
  • A score of 0.8 in one system does not inherently mean the same thing as 0.8 in another.
  • Rank positions are often safer inputs for an initial hybrid merge.

Fuse the two result lists with RRF

Reciprocal Rank Fusion assigns each document a contribution based on its position in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across all lists where d appears. The constant k reduces the impact of small rank differences near the top of a list.

For example, an application can request the top candidates from the S3 Vectors dense search path and the Quickwit BM25 sparse search path, deduplicate them by a stable document ID, then compute one fused rank. A document that appears near the top in both lists receives more support than one that appears only once at a low rank.

Choose a candidate depth that gives both retrievers a chance to contribute, then return the leading fused results. Keep the original dense rank, sparse rank, and fused score in diagnostic logs so relevance investigations remain explainable.

  • Use a stable canonical ID to identify the same document across both result sets.
  • Define rank as 1 for the first result, 2 for the second, and so on.
  • Compute: fused_score += 1 / (k + rank).
  • Start with one shared k value and evaluate it with representative queries before adding special cases.

Make hybrid retrieval observable and testable

RRF is intentionally simple, but it still needs evaluation against the queries your users actually ask. Build a small relevance set that includes semantic questions, exact-name lookups, identifier-heavy queries, short searches, and ambiguous terms. Record whether the desired document appears in the candidate set and how high it ranks after fusion.

Inspect disagreement as well as agreement. If BM25 repeatedly retrieves a document that dense search misses, that may reveal important exact terminology. If dense retrieval consistently finds useful paraphrases absent from BM25 results, that supports keeping the semantic path. The goal is not to force the lists to look identical; it is to use their differences productively.

Apply filters consistently before fusion whenever possible. A document excluded by tenant, access, language, or content-state rules should not be allowed back into the merged ranking. Consistent filtering also makes debugging much easier because both candidate lists are drawn from the same eligible corpus.

  • Log query text, filters, document IDs, per-source ranks, and final fused rank.
  • Evaluate candidate recall separately from final ranking quality.
  • Test duplicates, deleted content, and mismatched metadata keys across retrieval paths.
  • Re-run the evaluation set after changing embeddings, analyzers, document chunking, or query construction.