Why raw score addition is fragile

Dense-search scores describe a vector similarity relationship, while BM25 scores reflect term statistics and document length effects. Even when both result lists are relevant, their numeric ranges are not inherently comparable.

Adding raw scores can make one retriever dominate merely because its scoring scale is larger. That behavior may change as an index evolves, a query changes, or retrieval parameters are adjusted, making relevance difficult to reason about.

  • Do not assume a dense similarity score and a BM25 score share a common scale.
  • Avoid tuning a single fixed weight against unnormalized raw scores.
  • Keep each retrieval path independently observable before introducing fusion.

Use rank-based reciprocal rank fusion

A practical baseline is reciprocal rank fusion (RRF). Retrieve a bounded candidate list from dense search in S3 Vectors and another from Quickwit BM25, then assign each document a contribution based on its rank rather than its original score.

For each result list, add 1 divided by k plus the document rank to the document’s fused score. The constant k reduces the difference between adjacent top ranks; choose it as a configurable application setting and validate it with representative queries.

  • Request a candidate depth that is larger than the final number of results you plan to display.
  • Deduplicate documents by a stable document identifier before producing the final ranking.
  • Sum rank contributions when a document appears in both dense and sparse result lists.
  • Apply deterministic tie-breaking, such as a stable identifier, for repeatable responses.

Evaluate failures by query type

RRF is a strong operational starting point because it does not require score calibration, but it is still a ranking policy that should be evaluated. Build a small query set covering exact identifiers, product names, natural-language questions, abbreviations, and queries with uncommon terminology.

Inspect not only whether a desired document was returned, but also which retrieval path surfaced it and where it ranked after fusion. This makes it easier to identify cases where lexical precision, semantic recall, or the selected candidate depth needs adjustment.

  • Track overlap between dense and sparse candidate sets.
  • Review zero-result and low-overlap queries separately.
  • Compare fused rankings with each individual retrieval path.
  • Version fusion settings so ranking changes can be explained and rolled back.