Why raw-score blending is fragile

Dense retrieval and BM25 produce scores with different meanings and distributions. A dense similarity score reflects the relationship between query and embedding under a particular embedding model and metric. A BM25 score is driven by term frequency, document length normalization, and corpus-level term statistics.

Because those scales can shift with a model change, corpus growth, field configuration, or query type, a formula such as dense_score + bm25_score can quietly favor one retriever. Even weighting the terms does not remove the need to continually validate score calibration.

  • Embedding model changes can alter dense-score distributions.
  • BM25 scores vary with query terms and collection statistics.
  • Exact identifiers often create strong sparse matches.
  • Semantically similar wording can be found even when terms differ.

Fuse ranks with Reciprocal Rank Fusion

Reciprocal Rank Fusion, or RRF, combines result lists using position instead of raw score. Retrieve a candidate list from dense search and another from BM25, then assign each document a contribution based on its rank in each list. Documents returned by both methods rise naturally, while a high placement in either list remains valuable.

For each document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), summing across retrieval lists where d appears. The constant k reduces the difference between adjacent top ranks; choose it as a deliberate application setting and evaluate it on representative queries rather than assuming one value is universally correct.

  • Request the same candidate depth from each retriever as a simple starting point.
  • Deduplicate documents by a stable document identifier before presenting results.
  • Treat a missing document in one list as contributing zero from that list.
  • Keep source ranks and source scores in logs for diagnosis.

Implement the pattern at the retrieval boundary

In a Talqora-based retrieval path, issue a dense query against regional S3 Vectors and a sparse query through Quickwit BM25. Pass both ranked lists to a small fusion step in the application or retrieval service, then fetch or return the top fused documents. This keeps dense and sparse retrieval independently understandable while making the hybrid policy explicit.

Evaluate the fused ranking with queries drawn from real user tasks. Include acronym-heavy searches, exact-name lookups, paraphrased questions, multi-concept requests, and queries with ambiguous terminology. Review not only aggregate relevance but also cases where one retrieval path supplies a necessary result the other misses.

  • Version the embedding model, BM25 field choices, candidate depth, and RRF constant together.
  • Log the dense rank, sparse rank, and fused rank for selected results.
  • Use a stable corpus snapshot when comparing ranking-policy changes.
  • Apply metadata filters consistently to both retrieval branches when the product requires them.