Why raw dense and sparse scores should not be added casually

A dense retriever returns results based on proximity between embeddings. BM25 ranks documents from term overlap and weighting statistics. Even when both systems return numeric scores, those numbers do not necessarily share a scale, distribution, or interpretation.

Adding the two scores can make a ranking depend more on scoring behavior than on relevance. A change to embedding models, chunking, BM25 analysis, or corpus composition can alter score ranges and silently shift the balance between retrieval methods.

  • Dense retrieval can surface semantically related wording that does not share exact terms.
  • BM25 can strongly reward distinctive identifiers, product names, and exact phrases.
  • Score normalization requires assumptions that may not remain stable as the corpus changes.

Fuse ranked lists with reciprocal rank fusion

RRF starts by requesting a ranked candidate list from each retriever. For every document, it adds a contribution based on that document's position in each list. A document appearing in both lists receives contributions twice, while a document that ranks highly in one list can still remain competitive.

A common form is RRF(d) = sum over retrievers of 1 divided by k plus rank(d). The constant k reduces the difference between adjacent ranks near the top of a list. Treat k as a tuning parameter, and evaluate it against representative queries rather than assuming one value is universally best.

  • Retrieve a fixed top-N candidate set from dense search and from BM25.
  • Deduplicate documents using a stable document or chunk identifier.
  • Assign rank starting at 1 within each result list.
  • Sort the combined candidates by descending RRF score.

Make fusion observable and test it by query class

Keep retrieval diagnostics with each response during development: the dense rank, BM25 rank, fused rank, and which retrievers returned the item. These fields make it easier to identify whether a poor result came from candidate generation, fusion, chunk boundaries, or the source content itself.

Evaluate hybrid retrieval with query groups that reflect real usage. Exact-error-message and identifier queries are useful sparse-retrieval checks, while paraphrased questions and concept searches test dense retrieval. Include mixed queries because production traffic rarely separates cleanly into one category.

  • Compare dense-only, BM25-only, and fused rankings on the same labeled query set.
  • Inspect queries where one retriever finds a relevant item and the other does not.
  • Record candidate-list depth and fusion parameters alongside evaluation results.
  • Re-run the evaluation after changes to documents, chunking, embeddings, or analyzers.