Why raw dense and sparse scores should not be added

A dense search score describes similarity in an embedding space. A BM25 score is derived from term frequency, inverse document frequency, and document-length normalization. Even when both systems return larger-is-better values, the magnitude and distribution of those values are not inherently aligned.

Adding the scores directly can make one retriever dominate simply because its numerical range is wider. This behavior may change with a new embedding model, a different corpus, altered tokenization, or a BM25 configuration update. Score normalization can be useful, but it introduces assumptions that should be validated against relevance judgments.

A simpler starting point is to combine ranks rather than scores. Rank says only where a document appeared in each result list, avoiding a claim that a dense score of one value is equivalent to a BM25 score of another.

  • Dense retrieval is useful for semantic and paraphrased queries.
  • BM25 is useful for exact vocabulary, codes, names, and rare terms.
  • Raw score ranges from separate retrieval methods are not automatically comparable.

Fuse candidate lists with Reciprocal Rank Fusion

RRF assigns each document a contribution based on its position in every ranked list where it appears. For a document d, the fused score is the sum of 1 divided by k plus rank(d) across the lists. Lower rank numbers contribute more, so documents that rank well in either or both retrievers rise in the merged list.

In a Talqora-oriented retrieval pipeline, request a candidate list from dense search backed by regional S3 Vectors and a candidate list from sparse BM25 search backed by Quickwit. Deduplicate by a stable document or chunk identifier, calculate the RRF score for each candidate, sort descending, and send the highest-ranked items to the next stage.

The constant k reduces the gap between adjacent positions near the top of a list. Treat it as a relevance-evaluation parameter rather than a universal default. Likewise, choose candidate depths large enough to preserve useful overlap, but bounded enough to keep fusion and downstream processing predictable.

  • Retrieve top-N candidates independently from dense and BM25 search.
  • Use a stable ID to merge duplicate documents or chunks.
  • Compute RRF from positions, not raw retrieval scores.
  • Sort the deduplicated candidates by fused score before returning or reranking them.

Evaluate fusion by query shape, not only aggregate relevance

A hybrid strategy should be tested on queries that reflect production traffic. Include exact lookup queries, questions with domain jargon, paraphrases, ambiguous natural-language requests, and queries containing both a conceptual request and a literal constraint. These categories reveal whether fusion is recovering candidates that either individual retriever misses.

Inspect disagreements as carefully as aggregate metrics. A document found only by BM25 may indicate that identifiers or quoted phrases matter. A document found only by dense retrieval may reveal vocabulary mismatch. These cases can guide chunking, metadata filters, query rewriting, or later reranking work.

RRF is a candidate-generation method, not a guarantee that the first fused result is best. When an application needs stricter ordering, use the fused list as input to a separate reranking stage or application-specific business rules, while preserving the independent retrieval signals for debugging.

  • Track whether relevant items were found by dense search, BM25, or both.
  • Review zero-result and low-click queries separately.
  • Keep per-retriever ranks in logs for explainability.
  • Re-test after changing embeddings, analyzers, chunking, or corpus content.