Why raw dense and sparse scores should not be treated as equivalent

Dense retrieval returns documents based on proximity between embeddings. Sparse BM25 retrieval ranks documents from term-level evidence, including the presence and distribution of query terms. Both can be useful for the same query, but their scores represent different ranking systems.

A common implementation mistake is to add or average the two raw scores. That assumes the values share a stable meaning and range. In practice, changes to embedding models, index configuration, corpus composition, or BM25 settings can shift score distributions and make a fixed weighting difficult to reason about.

  • Dense retrieval can surface semantic matches when wording differs.
  • BM25 can strongly reward exact terms, identifiers, and rare phrases.
  • Raw score scales are not automatically comparable.
  • Rank positions are often a safer common signal for an initial hybrid design.

Fuse two ranked lists with reciprocal rank fusion

Run the same user query through the dense and sparse paths, then retain a ranked candidate list from each. Reciprocal rank fusion assigns every document a contribution based on its position in each list. Documents appearing near the top of either list receive more credit, while documents supported by both lists accumulate credit.

For each document d, calculate RRF(d) as the sum of 1 divided by k plus its rank in each result list. The constant k reduces the difference between adjacent top ranks and prevents a single rank-one result from overwhelming all other evidence. A value such as 60 is commonly used as a starting point, but it should be treated as a tunable application choice rather than a universal default.

  • Request a candidate depth from both dense and sparse retrieval paths.
  • Assign ranks starting at 1 within each returned list.
  • Sum the RRF contribution for documents present in one or both lists.
  • Sort by fused score and return the top results.

Make fusion observable and keep the first version simple

Store enough retrieval metadata to explain the merged ranking: whether a document came from dense search, BM25 search, or both; its rank in each list; and its final fused score. This makes relevance investigations concrete. A result that wins because of an exact identifier should look different from one that wins through agreement between lexical and semantic retrieval.

Start with unweighted RRF before introducing per-source weights, filters, rerankers, or query classifiers. Evaluate representative queries that include natural-language questions, product terminology, quoted phrases, error messages, codes, and names. If a later change is needed, adjust one variable at a time and compare the resulting ranked lists rather than relying only on aggregate impressions.

  • Log source membership and per-source ranks for each final result.
  • Use a small, reviewed query set to inspect ranking changes.
  • Check exact-match and terminology-heavy queries separately from semantic queries.
  • Add complexity only when observed retrieval failures justify it.