Treat dense and sparse scores as separate signals

A cosine-like dense similarity score and a BM25 score are produced by different models and statistics. Their ranges, distributions, and sensitivity to query length differ. Adding those raw values directly can make one retriever dominate simply because its numeric scale is larger, not because its result is more relevant.

Instead, issue the same user query to both retrieval paths: embed the query for regional S3 Vectors and send its text to Quickwit BM25. Ask each path for a bounded candidate list, then merge by a stable document or chunk identifier in the application layer. This preserves each retriever's strengths while avoiding an unsupported assumption about comparable scores.

  • Use one canonical ID for every indexed chunk in both systems.
  • Retrieve more than the final display count from each path so fusion has useful overlap and alternatives.
  • Keep metadata needed for filtering and presentation attached to the canonical ID or resolvable after fusion.
  • Log source rank and source membership for each fused result; these fields make relevance investigations much easier.

Use reciprocal rank fusion for a scale-free merge

RRF assigns a document a contribution based on its rank in each candidate list, then sums those contributions. For a document d, a common form is: RRF(d) = Σ 1 / (k + rank_i(d)). The sum runs across retrieval lists in which d appears, rank starts at 1, and k is a positive constant that reduces the advantage of being first by a small margin.

Because RRF consumes ranks instead of raw retrieval scores, it needs no score normalization between S3 Vectors and Quickwit. A chunk that appears near the top of both lists rises naturally, while a chunk found by only one retriever can still be retained. This is particularly useful when exact-token and semantic matches should both have a path into the final candidate set.

  • Start with the same candidate depth for dense and sparse retrieval to make early evaluation easier.
  • Use a fixed k consistently during an evaluation cycle; changing it changes ranking behavior.
  • Deduplicate before scoring so the same chunk contributes once per retrieval list.
  • Break equal fused scores deterministically, such as by best individual rank and then canonical ID.

Evaluate the fusion pipeline as a retrieval system

Build a small query set from real search intents, including exact identifiers, acronym-heavy queries, paraphrases, and multi-concept questions. For each query, record whether a relevant chunk is present in the dense list, the BM25 list, and the fused top results. These three views reveal whether a problem is candidate generation, fusion, or later answer generation.

Operationally, make filters and index versions explicit. A result can only be fused correctly when both candidate lists refer to the same corpus scope and compatible chunking scheme. If one path indexes a newer document revision or applies a different tenant filter, RRF may produce a plausible-looking but inconsistent ranking.

  • Measure recall at the candidate depth before judging final ranking quality.
  • Inspect queries where only one retriever finds the relevant chunk; they explain the value of hybrid retrieval.
  • Version embeddings, chunking rules, and sparse indexing configurations independently.
  • Apply authorization and tenant constraints before fusion, not after results are combined.