Why raw dense and sparse scores should not be added

A dense-search score reflects the relationship between query and document embeddings. A BM25 score is derived from term occurrences, term rarity, and document-length normalization. Even when both searches return relevant documents, their numeric outputs have different meanings and ranges.

Adding those values directly creates a hidden weighting decision. A change to embedding generation, corpus composition, analyzer settings, or query wording can change one score distribution without changing the other. Rank-based fusion avoids depending on a stable cross-system score scale.

  • Use dense retrieval for semantic similarity and paraphrased language.
  • Use BM25 for exact identifiers, names, error messages, and rare terms.
  • Treat each retriever's rank order as the portable signal for initial fusion.

Fuse two candidate lists with RRF

Run a dense query against the vector collection and a BM25 query against the sparse index. Request a bounded candidate list from each path, then deduplicate documents by a stable document ID. For every document returned by either path, calculate an RRF score from its position in each result list.

The common formula is RRF(d) = sum of 1 / (k + rank_i(d)), where rank_i starts at 1 for each retriever and k is a positive constant chosen by the application. A document appearing near the top of both lists receives more credit than one appearing only once or only at low rank.

  • Retrieve the same candidate depth from both paths as an initial baseline.
  • Use a stable ID shared by the dense record and the BM25 document.
  • Assign no contribution for a retriever where the document is absent.
  • Sort deduplicated documents by descending fused score before returning the final top results.

Evaluate fusion with queries that expose complementarity

RRF is easy to implement, but it should still be evaluated against representative traffic. Build a small labeled query set that includes exact-string lookups, natural-language questions, mixed queries containing both a concept and an identifier, and queries with ambiguous vocabulary. Compare dense-only, BM25-only, and fused rankings using the same relevance judgments.

Inspect failures rather than relying only on an aggregate metric. If exact-code queries are weak, BM25 candidate depth or text-field preparation may need attention. If paraphrased questions are weak, review chunking and embedding inputs. Fusion can increase coverage, but it cannot recover relevant material that neither retriever placed in its candidate set.

  • Log the source ranks that contributed to each fused result.
  • Keep candidate generation separate from any later reranking stage.
  • Test document-ID deduplication, especially when content is chunked.
  • Re-evaluate after corpus, chunking, embedding, or BM25 indexing changes.