Treat dense and sparse search as independent evidence

Start by sending the same user query through two retrieval paths. The dense path searches vector representations in regional S3 Vectors. The sparse path sends the textual query to Quickwit BM25. Each path returns an ordered candidate list with a stable document or chunk identifier.

The goal is not to decide which retriever is universally better. Instead, let each one contribute candidates where it has useful signals. A query such as a product code or error string may benefit from BM25’s exact-token matching, while a natural-language question may retrieve useful paraphrases through dense search.

  • Use the same canonical chunk ID in both indexes.
  • Request a bounded candidate list from each retriever.
  • Keep the original rank and score from each path for debugging.
  • Deduplicate candidates before presenting final results.

Fuse ranks instead of raw scores

RRF gives each document a contribution based on its position in each result list. For a document d, the fused score is the sum of 1 divided by k plus its rank in every list where it appears: RRF(d) = Σ 1 / (k + rank_i(d)). The constant k reduces the influence of small rank differences near the top of a list.

Because RRF consumes positions rather than retrieval scores, it does not require dense similarity values and BM25 scores to share a scale. Documents returned by both systems receive contributions from both lists, while documents found by only one system can still remain competitive.

  • Assign rank 1 to the first result in each list.
  • Choose one k value and keep it stable while evaluating changes.
  • Sum contributions for matching chunk IDs.
  • Sort candidates by fused score, then apply a deterministic tie-breaker.

Make the fusion layer observable and easy to tune

Implement fusion as a small application-layer step after retrieving the two ranked lists. Store enough information to inspect why a result won: dense rank, sparse rank, fused score, and whether the candidate appeared in one or both lists. This turns relevance investigations into concrete ranking questions instead of guesswork.

Evaluate with a representative set of real queries and judged relevant chunks. Look beyond a single aggregate metric: inspect queries with exact identifiers, short keyword searches, broad conceptual questions, and terminology that appears differently across documents. The most useful tuning changes are those supported by failure examples.

  • Log per-query candidate overlap between dense and sparse lists.
  • Review top fused results alongside their source ranks.
  • Compare candidate depths and k values using the same evaluation set.
  • Keep retrieval and fusion changes versioned so results remain reproducible.