Why raw-score blending is fragile

Dense retrieval and BM25 produce scores with different meanings. A dense score commonly represents semantic proximity under a chosen embedding model, while BM25 reflects term-based relevance derived from document and query statistics.

Adding or averaging those values assumes they share a stable scale. That assumption can break when the embedding model changes, document lengths shift, analyzer settings change, or a query contains an uncommon identifier such as an error code, SKU, or API parameter.

  • Do not treat a dense score of 0.8 as inherently comparable to a BM25 score of 0.8.
  • Avoid hard-coded weights unless they are validated against representative relevance judgments.
  • Expect score distributions to vary across query types and corpus updates.

Retrieve independently, then fuse by rank

A practical baseline is reciprocal rank fusion (RRF). Run a dense query against the S3 Vectors-backed retrieval path and a sparse BM25 query against Quickwit, then merge documents using their positions in each ranked list rather than their raw scores.

For each document, add 1 divided by k plus its rank for every list in which it appears. The constant k reduces the advantage of a single first-place result and makes the method less sensitive to small rank changes near the top.

  • Request a candidate set from both dense and sparse retrieval paths.
  • Assign ranks starting at 1 within each result list.
  • Compute: RRF(document) = sum of 1 / (k + rank).
  • Sort merged documents by the resulting RRF score and return the desired top results.

Use query-aware candidate depths and evaluate failures

Candidate depth is part of retrieval quality. If the application only fetches a few results from each method, a relevant document cannot be rescued during fusion when it falls just outside one list. Start with a depth larger than the final number of results, then tune it using real queries and relevance feedback.

Review queries by failure mode rather than only by an aggregate metric. Exact terms, product names, and identifiers often reveal the value of sparse retrieval. Natural-language questions, paraphrases, and conceptually related wording often reveal the value of dense retrieval.

  • Keep a small evaluation set containing both keyword-heavy and semantic queries.
  • Log the source ranks that contributed to each fused result.
  • Check whether relevant documents are missing from both candidate lists before changing fusion logic.
  • Re-evaluate after changes to embeddings, tokenization, document chunking, or indexed fields.