Why raw dense and sparse scores are difficult to combine

A dense-search score represents similarity in an embedding space. A BM25 score is based on term frequency, inverse document frequency, and document-length normalization. Even if both systems return a numeric relevance value, the numbers have different meanings and can change as data, query composition, or index settings change.

Adding the two scores directly, or applying a fixed threshold to each, can make one retriever dominate for reasons unrelated to result quality. For example, a query with a rare product code may favor BM25, while a natural-language question with paraphrased wording may benefit from dense retrieval.

  • Treat dense and BM25 scores as system-specific signals.
  • Request a ranked candidate list from each retriever.
  • Avoid assuming that a score of 0.8 in one list is equivalent to 0.8 in the other.

Fuse ranked lists with reciprocal rank fusion

RRF assigns each document a contribution based on its position in each result list. For a document d, calculate RRF(d) as the sum of 1 divided by k plus rank(d) for every list in which d appears. The constant k reduces the difference between adjacent top ranks and keeps the method from overreacting to a single position change.

In an API-first retrieval flow, query the dense index and the BM25 index independently, retain document identifiers and ranks, then merge the lists in application code. Documents returned by both paths accumulate contributions, while documents found by only one path can still be included.

  • Choose a candidate depth for both dense and sparse searches, such as the top N results from each.
  • Deduplicate by a stable document or chunk identifier before producing the final ranking.
  • Start with the same k value for both lists; tune only after reviewing representative queries.
  • Return the highest fused documents to downstream reranking or generation steps.

Evaluate the fusion policy using query categories

A single aggregate metric can hide where hybrid retrieval helps. Build a small evaluation set that separates exact-lookup queries, conceptual questions, mixed queries, and short ambiguous queries. Inspect whether the expected document enters the candidate set and whether its rank improves after fusion.

Also log retrieval provenance for each result: dense rank, BM25 rank, and fused rank. This makes failures explainable. If an exact identifier is consistently missed, investigate tokenization, field selection, or sparse-query construction. If semantically related material is absent, inspect chunking and embedding inputs rather than changing fusion weights first.

  • Compare dense-only, BM25-only, and fused rankings on the same labeled queries.
  • Review overlap between result lists; overlap is useful evidence, not a requirement.
  • Track query classes separately so tuning does not optimize only one search behavior.
  • Keep fusion deterministic to make regressions easier to reproduce.