Why raw score blending is risky

It is tempting to add a dense similarity score to a BM25 score, perhaps after choosing a weight for each. That approach assumes the scores have compatible meaning and stable ranges. In practice, a vector similarity value and a BM25 relevance value are produced by different models and ranking functions, so equal-looking numbers do not necessarily represent equal evidence.

Score distributions can also change with query length, corpus composition, analyzer settings, embedding model changes, or indexing updates. A fixed blending formula may work for a small test set while quietly favoring one retriever for another class of queries.

  • Dense similarity reflects geometric proximity in an embedding space.
  • BM25 rewards term matches using corpus-level term statistics.
  • Neither score should be assumed to be a calibrated probability of relevance.
  • Rank positions are often more comparable than raw scores.

Fuse rankings with reciprocal rank fusion

Reciprocal rank fusion (RRF) combines ordered result lists rather than their original scores. For each document, add 1 divided by k plus its rank for every list in which it appears. The document is then ordered by its accumulated RRF score. The constant k dampens the advantage of appearing at the very top of one list.

For a query, request a candidate list from dense retrieval and a candidate list from BM25. Identify documents with a stable shared ID, compute the RRF total in the application, deduplicate by that ID, and return the highest fused results. A document found by both methods receives evidence from both rankings; a document found by only one can still survive if it ranks highly.

  • Formula: RRF(d) = Σ 1 / (k + rank_i(d)).
  • Use one-based ranks: the first item in a list has rank 1.
  • Choose a single k initially and evaluate it against representative queries.
  • Keep the source ranks during debugging so fused results are explainable.

Make the fusion layer observable and testable

Treat fusion as retrieval logic, not as a cosmetic post-processing step. Log the query, document ID, dense rank, BM25 rank, fused rank, and whether the document appeared in one or both lists. These fields make it possible to diagnose a poor result: perhaps the identifier was only visible to BM25, or perhaps a paraphrase was only surfaced by dense retrieval.

Build an evaluation set that includes semantic questions, exact product names, internal codes, acronyms, misspellings, and multi-part questions. Compare dense-only, BM25-only, and RRF results using judgments from the people who know what a useful answer looks like. If downstream generation is involved, evaluate retrieval independently before attributing answer quality to the model.

  • Use the same document identifiers across the dense and sparse indexes.
  • Set candidate-list depth deliberately; fusion cannot promote documents neither retriever returned.
  • Inspect overlap rates between dense and BM25 lists by query class.
  • Re-evaluate after changes to embeddings, text processing, or corpus contents.