Why raw-score blending is fragile

A dense-search score and a BM25 score are produced by different models and different statistical assumptions. Even if both are returned as numbers where a larger value appears better, their ranges and distributions can vary by query, corpus, analyzer configuration, embedding model, and similarity metric.

Adding the two scores directly introduces a hidden tuning problem: a weight that works for one query class may suppress useful results for another. A query containing a product code may need strong lexical matching, while a natural-language question may benefit more from semantic neighbors. Rank-based fusion avoids treating either system's numeric score as a universal relevance scale.

  • BM25 scores depend on term frequency, document frequency, and field analysis.
  • Dense scores depend on the embedding representation and chosen vector similarity.
  • Per-query normalization can be unstable when one retriever returns a narrow or unusually broad score distribution.

Fuse candidate lists with Reciprocal Rank Fusion

Reciprocal Rank Fusion, or RRF, assigns a document a contribution based on its position in each ranked list rather than its original score. For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)), summing across the retrieval lists where d appears. Lower rank numbers are better, and k is a positive constant that reduces the difference between nearby positions.

For each query, retrieve a bounded candidate list from S3 Vectors and another from Quickwit BM25. Deduplicate by a stable document or chunk identifier, calculate the RRF total, then sort descending by that total. A result appearing near the top of both lists receives a stronger combined signal; a result found by only one retriever can still remain competitive.

  • Use the same stable ID in dense and sparse indexes so duplicate chunks can be merged.
  • Keep rank positions one-based: the first result has rank 1.
  • Choose candidate depths deliberately; fusion cannot promote a document that neither retriever returned.
  • Treat k as an evaluation parameter, not as a universal constant.

Make the fusion stage observable and testable

Log the source rank and RRF contribution for every fused result. These fields make it possible to explain whether a result won because both retrievers agreed, because BM25 strongly matched a rare term, or because dense search supplied a semantic match missing from the lexical list. They also help identify failures such as inconsistent IDs, mismatched filtering, or an overly shallow candidate pool.

Evaluate the fused ranking against representative queries before changing defaults. Include exact-name lookups, error messages, short ambiguous queries, paraphrases, and domain vocabulary. Review not only aggregate relevance measures but also result-set overlap: low overlap is not automatically bad, but it indicates that the two retrieval methods are contributing different candidates.

  • Apply equivalent tenant, authorization, and document-state filters before fusion.
  • Record dense rank, BM25 rank, fused rank, and the final document ID.
  • Inspect queries where only one retriever contributes to top results.
  • Re-test fusion when changing analyzers, chunking, embedding models, or index contents.