Why combine candidate sets instead of raw scores?

Dense and sparse systems usually produce scores with different meanings. A vector similarity score reflects geometric proximity in an embedding space, while BM25 is driven by term frequency, document frequency, and length normalization. Treating those values as directly comparable can create unstable ranking behavior.

Candidate union avoids that assumption. Run a dense query against the vector index and a sparse query against the BM25 index, then deduplicate the resulting document identifiers. The combined set gives a downstream ranking step access to both semantic matches and exact lexical matches.

  • Dense retrieval can help when a user’s wording differs from the document’s wording.
  • BM25 can preserve matches for product codes, names, error messages, and rare terms.
  • Deduplication should use a stable canonical document or chunk identifier.
  • Keep the retrieval paths independently observable so missing or weak candidates are easier to diagnose.

Use Reciprocal Rank Fusion as a practical first combiner

Reciprocal Rank Fusion combines ranked lists rather than their raw scores. For each document, add a contribution based on its position in each list: 1 divided by k plus the document rank. The constant k reduces the influence of small rank differences near the top of a list.

If a document appears in both the S3 Vectors result list and the Quickwit BM25 result list, it receives contributions from both. If it appears in only one list, it can still rank well when it is near that list’s top. This makes RRF a useful baseline when score calibration has not been established.

  • Choose a single rank convention, such as rank 1 for the first result.
  • Apply the same candidate depth to both paths at first, then adjust using relevance evaluation.
  • Use a fixed k during an experiment so ranking changes are attributable to retrieval changes.
  • Store each document’s source ranks alongside the fused result for debugging.

Evaluate failures by query type, not only aggregate relevance

A hybrid design is most useful when evaluated against representative query classes. Build a small labeled set that includes exact identifier searches, natural-language questions, acronym-heavy queries, and cases where relevant documents use different wording. Compare dense-only, BM25-only, and fused rankings on the same set.

Reviewing result provenance is as important as reviewing the final order. For a failed query, determine whether the relevant item was absent from both candidate lists, present but ranked too low, or lost during fusion. Each failure points to a different intervention: indexing and chunking, query construction, candidate depth, or ranking policy.

  • Track whether relevant results were supplied by dense retrieval, sparse retrieval, or both.
  • Inspect duplicate handling when multiple chunks belong to the same source document.
  • Test short keyword queries separately from full-sentence queries.
  • Treat RRF as a baseline that can be revised after collecting reliable relevance judgments.