Why dense and sparse scores should not be compared directly

Dense retrieval returns results based on the relationship between embedding vectors. Depending on the similarity measure and index implementation, its scores may represent cosine similarity, distance, or another internal ranking value. BM25, meanwhile, scores documents from term frequency, inverse document frequency, and document-length normalization.

Those values are useful within their own retrieval systems, but they do not automatically form a shared relevance scale. A dense score of 0.72 and a BM25 score of 8.4 do not provide enough information to conclude which document is more relevant. Adding, averaging, or thresholding raw scores can make ranking behavior difficult to reason about.

  • Embedding models can change the distribution of dense scores.
  • BM25 scores vary with corpus composition and query terms.
  • Different result sets may have different score ranges for the same query.
  • Rank positions are easier to interpret across retrieval methods than raw scores.

Apply reciprocal rank fusion to the two candidate lists

RRF assigns each document a contribution based on its position in each ranked list. For a document d, calculate RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank in retrieval list i and k is a positive constant. Documents returned by both dense and BM25 retrieval receive contributions from both lists.

The constant k reduces the difference between adjacent top positions and prevents a single rank-one placement from overwhelming all other evidence. A value such as 60 is commonly used as an initial convention, but it should be treated as a configuration choice rather than a universal optimum. Keep the dense and sparse candidate depths explicit so the fusion stage has enough documents to reconcile.

  • Retrieve a bounded candidate list from dense search.
  • Retrieve a bounded candidate list from BM25 search.
  • Deduplicate candidates by stable document identifier.
  • Sum the RRF contribution for every list in which a document appears.

Make fusion observable before making it more complex

Log the dense rank, BM25 rank, and final fused rank for returned documents. These fields explain whether a result won through semantic agreement, exact lexical evidence, or support from both paths. They also reveal queries where one retriever contributes almost nothing, which may indicate a query-analysis, indexing, or content-quality issue.

Start with unweighted RRF, then evaluate representative query sets that include identifiers, quoted language, broad conceptual questions, and mixed natural-language queries. If later evidence shows that one retrieval mode deserves more influence for a defined query class, introduce weights deliberately and retain the same rank-level observability.

  • Track candidate-set overlap between dense and BM25 retrieval.
  • Inspect documents that rank highly in only one retrieval path.
  • Evaluate failures by query intent, not only aggregate relevance judgments.
  • Version fusion settings alongside embedding and indexing changes.