Why raw-score blending is fragile

A dense-search score and a BM25 score are produced by different ranking functions. Their ranges, distributions, and sensitivity to query length can differ substantially. Adding them together with a fixed weight may work for one query set while producing unstable rankings for another.

For example, an exact product code may be strongly favored by BM25, whereas a natural-language question may have stronger semantic matches in dense retrieval. A score threshold that is meaningful in one system is not automatically meaningful in the other.

  • BM25 depends on term frequency, document frequency, and length normalization.
  • Dense ranking depends on the embedding model and its similarity measure.
  • Score magnitudes can change after corpus, analyzer, or embedding updates.
  • A merged ranking should preserve useful evidence from both retrieval methods.

Fuse ranked lists with Reciprocal Rank Fusion

Reciprocal Rank Fusion, or RRF, combines lists by rank position instead of raw score. Run the same query through dense retrieval and BM25 retrieval, then assign each returned document a fusion contribution based on its position in each list.

For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) across every list where d appears. The constant k reduces the impact of small rank differences near the top of a list. A commonly used starting value is 60, but it should be treated as a tunable parameter rather than a universal rule.

  • Retrieve a bounded candidate list from regional S3 Vectors for the dense path.
  • Retrieve a bounded candidate list from Quickwit BM25 for the sparse path.
  • Deduplicate documents by a stable document identifier.
  • Sort the union by its summed RRF score and return the top results.

Make fusion observable and tune it with real queries

Start by logging each document’s dense rank, BM25 rank, fused rank, and final selection status. These fields make it possible to inspect whether a result won because both systems agreed or because one retriever contributed a uniquely valuable match.

Evaluation should include queries that reflect the intended workload: exact identifiers, error messages, short keyword searches, paraphrased questions, and multi-concept requests. Review not only aggregate relevance judgments but also failure cases where an exact-match result or a semantic result disappears after fusion.

  • Compare dense-only, BM25-only, and fused result sets for the same evaluation queries.
  • Test candidate depths separately from the final number of returned results.
  • Use stable IDs and versioned documents so duplicates and stale content can be diagnosed.
  • Re-evaluate after changing embeddings, tokenization, document chunking, or corpus composition.