Why raw-score blending is fragile

Dense retrieval ranks vectors by a similarity or distance measure, while BM25 ranks documents from term-frequency and corpus-statistics signals. Even when both systems return numeric scores, identical-looking numbers do not imply identical relevance. A dense score can shift when embeddings or similarity settings change; BM25 scores can shift as the indexed corpus changes.

A weighted formula such as dense_score + sparse_score introduces a calibration problem. Before choosing weights, an engineer must establish how scores are normalized, how stable that normalization remains over time, and whether the two candidate lists have similar score distributions. Those questions can be harder than the retrieval query itself.

  • Dense search helps with semantic matches and alternate phrasing.
  • BM25 helps when exact terms, identifiers, or rare tokens matter.
  • Raw scores should not be assumed comparable across retrieval methods.
  • Rank positions are easier to interpret than unrelated score scales.

Fuse ranked lists with RRF

RRF assigns each document a contribution based on its position in each ranked list. For a document d, the fused score is the sum of 1 / (k + rank_i(d)) across the lists where d appears. The constant k reduces the impact of small differences near the top of a list and keeps the method focused on consistently high-ranking documents.

In a Talqora retrieval path, request a candidate set from dense search over regional S3 Vectors and another from Quickwit BM25. Preserve document identifiers, ranks, and any metadata needed for filtering or display. Then merge candidates by identifier and calculate the RRF score in the application or ranking layer.

  • Choose a candidate depth for each retriever, such as the top N results.
  • Number ranks starting at 1 for each individual result list.
  • Sum 1 / (k + rank) for every list that contains the document.
  • Sort by fused score, then apply a deterministic tie-breaker such as document ID.

Tune the candidate set, then evaluate failures

RRF has few parameters, but candidate depth still matters. If dense and sparse retrieval each return too few candidates, a relevant document cannot be rescued by fusion because it never enters either list. If they return excessively deep lists, low-ranked noise may increase processing cost without improving the final page of results.

Evaluate hybrid retrieval with a query set that represents real search behavior: natural-language questions, exact product names, error messages, part numbers, and short ambiguous queries. Inspect misses rather than relying only on an aggregate metric. A query that requires an exact identifier may expose a sparse-retrieval issue, while a paraphrased question may reveal a dense-retrieval or chunking issue.

  • Log each document's dense rank, BM25 rank, and final fused rank.
  • Keep filters consistent across both candidate sources when possible.
  • Test several candidate depths and k values against labeled or reviewed queries.
  • Treat fusion as one stage: document chunking, metadata design, and query construction still shape recall.