Why raw-score blending is fragile

Dense retrieval and BM25 both return ranked documents, but their scores have different meanings. A dense-search score reflects proximity in an embedding space, while a BM25 score is driven by term frequency, document frequency, and length normalization. Even when both lists contain relevant items, a score of 0.8 from one system is not inherently comparable to a score of 8 from the other.

A common first attempt is to add weighted scores: dense_score × alpha plus bm25_score × beta. This can work only when score distributions are well understood and remain stable across query types, indexes, and configuration changes. In practice, a query with exact identifiers may favor lexical matching, while a broad conceptual query may favor dense retrieval. One fixed score scale can be difficult to maintain.

  • Dense search can recover semantically related wording.
  • BM25 can strongly reward exact terms, names, codes, and rare phrases.
  • Raw scores are model- and engine-specific signals, not universal relevance units.

Fuse positions with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in every result list. For a document d, the fused value is the sum of 1 divided by k plus its rank in each list where it appears. The constant k reduces the gap between top ranks and lower ranks, preventing a single list from dominating solely because of rank position.

For example, if a document ranks 2nd in the dense list and 5th in the BM25 list, its fused score is 1/(k+2) + 1/(k+5). A document appearing in both lists is rewarded for agreement. A document that appears only in one list can still be returned, which preserves useful semantic-only or lexical-only matches.

  • Retrieve a bounded candidate list from dense search and another from BM25.
  • Use one-based ranks when calculating the RRF contribution.
  • Deduplicate by a stable document or chunk identifier before returning results.
  • Sort by fused score, then apply a deterministic tie-breaker such as document ID.

Make fusion observable and query-aware

Start with a simple, explicit pipeline: obtain dense candidates from the regional S3 Vectors side, obtain sparse candidates from Quickwit BM25, and fuse the two ranked lists in the application layer. Log each candidate’s dense rank, BM25 rank, and final fused score. These fields make it possible to explain why a result surfaced and to inspect whether one retriever is unexpectedly absent from top results.

Evaluate using a representative set of real information needs, especially queries containing product names, identifiers, abbreviations, and paraphrased questions. Review not only whether the best answer appears, but also whether the top results contain duplicates, near-duplicates, or overly broad chunks. RRF is a strong baseline because it reduces score-calibration work; it does not replace careful chunking, metadata filtering, or relevance evaluation.

  • Keep dense and sparse retrieval limits configurable per query class.
  • Record rank provenance for debugging and relevance reviews.
  • Apply metadata and access filters consistently to both candidate lists.
  • Use judged queries to validate changes to chunking, analyzers, embeddings, or fusion settings.