Why raw dense and sparse scores are difficult to blend

A dense retrieval system and a BM25 index may both return a number called a score, but those numbers usually represent different computations. Dense similarity depends on the embedding model, vector normalization, and distance metric. BM25 depends on term frequency, document frequency, field length, and its tuning parameters.

Adding the two scores directly creates an implicit claim that one unit of dense similarity should equal one unit of BM25 relevance. That claim is rarely stable across queries. A query containing a rare product code may produce a sharply peaked BM25 distribution, while a broad natural-language question may produce much flatter lexical scores and more useful dense results.

  • Score ranges can vary by query, not only by index.
  • A change in embedding model can alter dense-score behavior.
  • BM25 score magnitudes depend on corpus statistics and query terms.
  • Raw weighted sums require calibration and ongoing evaluation.

Fuse ranks with reciprocal rank fusion

RRF starts by asking each retriever for its top results. For every document, assign a contribution based on its rank in each list, then sum the contributions. A common form is RRF(d) = Σ 1 / (k + rankᵢ(d)), where rankᵢ(d) is the document's one-based position in retriever i and k is a positive constant.

Because the formula uses positions, it does not require dense and BM25 scores to be comparable. A document that ranks well in both lists rises naturally. A document found only by one retriever can still appear, which preserves the complementary recall of dense search and lexical search.

  • Retrieve a candidate list from dense search and another from BM25.
  • Use a consistent document identifier to merge duplicate results.
  • Treat a missing document as contributing zero from that retriever.
  • Choose k as a tunable smoothing parameter and validate it on representative queries.

Implement the fusion layer as a small, observable service step

In an API-first architecture, the application can query the dense and sparse retrieval paths, merge their result sets, and return the fused ranking to the caller. Keep the original rank and score from each source in the response or internal logs. Those fields make it possible to investigate why a result appeared and whether one retrieval path is dominating particular query classes.

Start with equal treatment of the two ranked lists before introducing weights. If evaluation shows that one source should have more influence for a known workload, use a weighted RRF variant rather than mixing raw scores: score(d) = Σ wᵢ / (k + rankᵢ(d)). Evaluate changes using labeled relevance judgments, query slices such as identifiers versus conceptual questions, and failure cases from production feedback.

  • Log query type, source ranks, fused rank, and selected document IDs.
  • Deduplicate by canonical ID before presenting final results.
  • Set candidate depths high enough that fusion has alternatives to combine.
  • Evaluate dense-only, BM25-only, and fused rankings side by side.