Why raw-score blending is fragile

A dense-search score and a BM25 score are produced by different retrieval methods. Their numeric ranges, distributions, and sensitivity to query length can differ, so adding them together with a fixed weight can create unexpected ranking changes.

The problem becomes especially visible across varied traffic. A short query containing an exact product code may benefit strongly from sparse retrieval, while a natural-language question may have useful semantic matches that do not share many terms with the query.

  • Do not assume a score of 0.8 from one retriever means the same thing as 0.8 from another.
  • Avoid choosing a dense-versus-sparse weight solely from a handful of example queries.
  • Treat each retriever’s ordered result list as the more portable signal.

Run both retrieval paths and apply reciprocal rank fusion

A simple application-layer pattern is to issue the same user query to the dense and sparse paths, collect a bounded ranked list from each, and merge the lists with reciprocal rank fusion (RRF). Talqora’s API-first model fits this pattern because the application can own the retrieval orchestration and final ranking policy.

For each document, RRF sums 1 divided by k plus the document’s rank for every list in which it appears. The constant k reduces the influence of small rank differences near the top of a list. Documents returned by both methods receive support from both rankings, while a strong result from only one method can still be retained.

  • Request a modest candidate set from regional S3 Vectors for dense retrieval.
  • Request a similarly bounded candidate set from Quickwit BM25 for sparse retrieval.
  • Use one-based ranks and deduplicate documents by a stable document identifier.
  • Sort by the summed RRF score, then fetch or return the selected documents.

Make the fusion policy observable and easy to revise

Log the query, the rank contributed by each retrieval path, the fused rank, and the selected document identifiers. This record makes relevance review concrete: a reviewer can see whether an item won because of dense retrieval, BM25, or agreement between both.

Start with one fusion rule for all queries before adding exceptions. If evaluation later shows a consistent pattern—such as identifier-heavy queries needing stronger exact-match treatment—introduce a clearly defined routing or weighting rule and compare it against the baseline on a representative query set.

  • Keep dense and sparse result counts configurable rather than hard-coded.
  • Review queries where the two ranked lists have little overlap.
  • Evaluate changes with judged queries that include semantic questions, exact terms, and mixed queries.
  • Version the fusion settings so ranking changes can be traced and rolled back.