Treat dense and sparse search as complementary signals

A user asking "how do I rotate an API credential?" may not use the same language as documentation titled "regenerate access keys." Dense retrieval can bridge that vocabulary gap because it compares vector representations of the query and documents.

Conversely, a query such as "ERR_CONN_142" or "SKU-A19-XL" may depend on exact tokens. Sparse retrieval with BM25 can preserve those lexical signals, including terms whose meaning may not be reliably represented by an embedding. Running both retrieval modes gives the ranking layer two different views of relevance.

  • Use dense retrieval for paraphrases, concepts, and natural-language questions.
  • Use BM25 for identifiers, quoted phrases, rare terms, and terminology.
  • Keep the same document IDs across both indexes so results can be merged reliably.

Fuse ranks instead of directly adding scores

Dense similarity values and BM25 scores are not naturally comparable. Their ranges and distributions can change with the embedding model, corpus, query length, analyzer settings, and retrieval configuration. Adding raw scores can accidentally make one retriever dominate for numerical reasons rather than relevance.

Reciprocal Rank Fusion (RRF) avoids that calibration problem by using positions in each result list. For every document, add 1 divided by k plus its rank from each list where it appears. The constant k reduces the influence of a single top position; a commonly used starting point is 60, but it should be treated as a tunable application choice rather than a universal rule.

  • Retrieve a candidate list from dense search and another from BM25.
  • Assign ranks starting at 1 within each list.
  • Compute RRF(document) = Σ 1 / (k + rank).
  • Deduplicate by document ID, sort by fused score, and return the top results.

Make fusion observable and test it with real queries

A fused ranking is only useful if it can be inspected. Record whether each returned document came from dense retrieval, BM25, or both, along with its individual ranks and final fused score. This makes it easier to diagnose cases where an exact-match result is missing, a semantic near-match is over-promoted, or the candidate depth is too shallow.

Build a small evaluation set from representative traffic: support questions, documentation searches, part numbers, abbreviations, and queries that have no answer. Review top-k results before and after fusion. If exact-token queries are weak, inspect tokenization and document text; if paraphrases are weak, inspect chunk boundaries and embedding input. Fusion improves combination, but it cannot repair missing or poorly indexed content.

  • Log per-retriever rank and inclusion for every fused result.
  • Evaluate recall at a fixed candidate depth before tuning fusion.
  • Include zero-result and ambiguous queries in relevance reviews.
  • Version retrieval settings so ranking changes can be compared over time.