Why raw-score blending is fragile

Dense retrieval ranks documents by vector similarity, while BM25 ranks them using term-frequency and document-statistics signals. Even when both systems return a numeric score, the magnitude, distribution, and meaning of those scores differ.

A weighted formula such as dense_score × 0.7 + bm25_score × 0.3 can work only after careful normalization and ongoing validation. Changes in embedding models, document lengths, index settings, or query mix can alter score distributions and quietly change the behavior of that formula.

  • Dense search is often useful when a query and document use different but related language.
  • BM25 is often useful for exact terms, identifiers, error messages, names, and rare vocabulary.
  • Raw scores should not be assumed to share a common scale.
  • Rank positions are easier to compare across retrieval methods than raw scores.

Fuse ranked lists with RRF

Reciprocal Rank Fusion assigns each document a contribution based on its position in each ranked result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus the document’s rank, across all lists where it appears. The constant k reduces the difference between nearby ranks and prevents rank one from dominating too aggressively.

For example, retrieve a candidate list from regional S3 Vectors and another from Quickwit BM25. If a document is ranked highly by both, its contributions add together. If it appears in only one list, it can still be returned, but documents with agreement across retrieval methods are naturally promoted.

  • Use one-based ranks: the first result has rank 1.
  • A common starting point is k = 60, then validate it against representative queries.
  • Deduplicate documents by a stable document identifier before producing the final ranking.
  • Keep the source ranks with each candidate to make ranking behavior inspectable.

Choose candidate depth and evaluate failures

RRF can only promote documents that appear in at least one input list. Request enough candidates from both dense and sparse retrieval to give fusion meaningful choice. The right depth depends on corpus size, latency requirements, and how many final results the application displays, so it should be selected through evaluation rather than copied from a generic recipe.

Build a small query set from real user intents, including exact identifiers, broad conceptual questions, ambiguous phrasing, and terminology that changed over time. For each query, compare dense-only, BM25-only, and fused results. Review not only whether a relevant document appears, but whether it reaches the positions users actually inspect.

  • Log the query, source rank, fused rank, and document ID for debugging.
  • Inspect cases where one retriever finds a relevant result that fusion pushes down.
  • Test candidate-depth changes separately from changes to k.
  • Treat retrieval evaluation as a recurring process when content, embeddings, or query behavior changes.