Why raw-score blending is fragile

Dense retrieval and BM25 answer different questions. Dense search ranks documents by proximity in an embedding space, while BM25 ranks them from term statistics and document length. Both produce ordered results, but their numeric scores have different meanings.

Adding or averaging those scores can make retrieval behavior sensitive to changes that are unrelated to relevance. A new embedding model, a different corpus, altered chunk sizes, or updated BM25 settings can shift score distributions. A blend that worked for one configuration may need retuning after any of those changes.

  • A dense score is not automatically comparable to a BM25 score.
  • Score ranges can vary by query, corpus, and model configuration.
  • Ranking position is usually more stable than an uncalibrated raw score.
  • Use score blending only when scores have been deliberately calibrated.

Fuse candidate lists by reciprocal rank

A straightforward alternative is reciprocal rank fusion (RRF). Retrieve a top-k list from dense search and a top-k list from BM25, then assign each document a contribution based on its position in each list. Documents appearing in both lists accumulate evidence, while documents that rank highly in either list can still surface.

A common form is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank of document d in retrieval list i and k is a positive constant. The constant softens the advantage of the first few positions, making the fusion less dominated by a single rank-one result.

  • Run dense retrieval against the vector index.
  • Run BM25 retrieval against the sparse index.
  • Deduplicate candidate IDs across both result lists.
  • Sum each candidate's reciprocal-rank contributions and sort by the total.

Make fusion observable and testable

For Talqora's architecture, dense candidate generation can use regional S3 Vectors, while sparse candidate generation can use Quickwit BM25. Keep the two result lists visible in logs or evaluation output before looking only at the fused order. This makes it possible to see whether a relevant result was found by dense search, sparse search, or both.

Evaluate with queries that represent the language your users actually write. Include exact identifiers, product names, error messages, abbreviations, paraphrases, and multi-concept questions. When a result is poor, first identify whether the failure came from candidate generation, fusion, chunking, or the source content itself.

  • Record the dense rank, BM25 rank, and final fused rank for inspected results.
  • Choose candidate-list depths large enough to preserve useful overlap.
  • Test changes with a fixed query set before changing several retrieval settings at once.
  • Treat RRF parameters as evaluation inputs, not permanent defaults.