Why raw-score blending is fragile

A dense retrieval system returns results according to vector similarity or distance. Quickwit BM25 returns results according to term-frequency and corpus-based relevance signals. Both are useful rankings, but the magnitude of one score does not establish an equivalent magnitude in the other system.

For example, a query containing an exact incident code may deserve strong lexical matching, while a natural-language question may benefit from semantic neighbors. Adding raw scores requires normalization assumptions that can change as content, embeddings, analyzers, or query patterns change. Those assumptions are often difficult to inspect and maintain.

  • Dense and sparse scores may use different ranges and distributions.
  • Score behavior can vary by query, not only by index.
  • A fixed weighted sum can over-favor one retriever unexpectedly.
  • Rank order is usually easier to reason about than score magnitude.

Fuse ranks with a simple RRF calculation

In Talqora, a hybrid retrieval path can query regional S3 Vectors for dense candidates and Quickwit BM25 for sparse candidates. Keep the two calls independent, request a bounded candidate list from each, and merge their document identifiers in an application-side fusion step.

RRF assigns each document a contribution based on its position in each ranked list: 1 divided by k plus the rank. Sum that contribution across lists. A document returned by both retrievers rises naturally, while a highly ranked result from only one retriever can still be retained.

  • Use one-based ranks: the first result has rank 1.
  • Calculate: RRF(document) = sum of 1 / (k + rank).
  • Choose a constant k and keep it stable while evaluating changes.
  • Deduplicate by a canonical document or chunk identifier before returning results.

Build an evaluation loop around candidate quality

Start with a small set of representative queries: exact identifiers, short keyword searches, paraphrased questions, multi-concept requests, and queries that should return nothing. For each query, record whether useful content appears in the fused top results and whether an important lexical or semantic result was lost.

RRF is intentionally simple, but it still has operational choices. Candidate depth affects recall, chunk identity affects deduplication, and metadata filters must be applied consistently to both retrieval paths. Log the per-retriever rank and the final fused rank so that relevance investigations remain explainable.

  • Apply the same tenant, permission, and content-state filters to both searches.
  • Fetch enough candidates that fusion has meaningful overlap and alternatives.
  • Store dense rank, BM25 rank, and fused rank in retrieval diagnostics.
  • Use judged queries to compare configuration changes before rollout.