Why raw-score blending is fragile

A tempting hybrid formula is to add a dense score to a BM25 score, perhaps after multiplying one side by a weight. That can work in a narrowly controlled system, but the two scores usually have different meanings and distributions. BM25 reflects term statistics and document length effects; dense similarity reflects the relationship between an embedded query and embedded content.

Score ranges can also vary across queries. An exact, rare identifier may create a sharp BM25 leader, while a broad natural-language query can produce a flatter sparse ranking. Dense retrieval has its own behavior based on the embedding model and the corpus. A fixed score weight can therefore overreact to one retrieval path on some queries and underweight it on others.

  • Do not assume a BM25 score of one value is equivalent to a vector similarity score of the same value.
  • Avoid choosing blend weights solely from a few hand-picked queries.
  • Treat ranking order as the more portable signal when score semantics differ.
  • Keep dense and sparse result lists available for inspection during relevance debugging.

Apply reciprocal rank fusion to two candidate lists

Run the same user query through the dense path and the Quickwit BM25 path, requesting a candidate depth larger than the final number of results you intend to show. Normalize result identities before merging: every chunk or document should have a stable canonical ID, even if the dense and sparse representations are stored separately.

For each unique result, calculate an RRF score by summing 1 divided by k plus its one-based rank for every list in which it appears. In symbols: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the influence of small rank differences near the top of a list. Sort by the summed RRF score, then return the top results.

  • Use one-based ranks: the first result has rank 1.
  • Deduplicate by canonical document or chunk ID before returning results.
  • Use the same candidate depth for both paths as an initial operational baseline.
  • Preserve per-source ranks in logs so a fused result can be explained later.

Make fusion observable and tune it with judged queries

RRF is a robust starting point, not a substitute for evaluation. Build a small query set that reflects production traffic: conceptual questions, exact-name lookups, identifier searches, short ambiguous queries, and domain terminology. For each query, record which result is useful at the retrieval stage, before any downstream answer-generation step can hide retrieval mistakes.

Review dense-only, BM25-only, and fused rankings side by side. If exact strings are routinely missed, inspect tokenization, field selection, and the text supplied to BM25. If semantically relevant material is absent, inspect chunk boundaries and the content used to create embeddings. Only after those fundamentals are sound should you vary candidate depth or the RRF constant.

  • Log query type, source ranks, fused rank, and selected result IDs.
  • Separate retrieval relevance from presentation or generation quality.
  • Test changes against a fixed judged-query set before rollout.
  • Investigate missing candidates before adjusting fusion parameters.