Start with two independent retrieval paths

A hybrid retrieval request should begin as two retrieval operations rather than one blended assumption. Send an embedding of the user query to the dense path, backed by S3 Vectors, and send the original text query to the sparse BM25 path, backed by Quickwit. Each path returns a ranked list of document identifiers and its own score.

Keeping the paths independent is important because the two scoring systems do not share a natural scale. A dense similarity score and a BM25 score represent different ranking signals; comparing their raw numeric values directly can produce accidental bias. The first goal is candidate generation, not immediate score arithmetic.

  • Use the original user query for BM25, preserving identifiers, quoted phrases, and uncommon terms.
  • Use a consistently generated query embedding for dense retrieval.
  • Request a bounded candidate set from each path, such as the top N document IDs.
  • Store document IDs and retrieval-source metadata alongside each candidate.

Fuse ranks instead of raw scores

A practical first fusion method is Reciprocal Rank Fusion (RRF). For each document, add a contribution based on its position in every result list where it appears. Documents that rank well in both lists rise naturally, while a strong result from only one method can still remain visible.

RRF is useful because it operates on rank positions rather than assuming that dense and BM25 scores are calibrated. A common form is: fused_score = sum of 1 divided by (k plus rank). The constant k reduces the difference between nearby positions and can be selected through offline relevance testing rather than intuition alone.

  • Deduplicate candidates by document ID before producing the final ranking.
  • Record each document's dense rank, BM25 rank, and fused rank for debugging.
  • Choose candidate depths large enough to allow overlap, but small enough to control downstream work.
  • Evaluate fusion changes with representative queries containing both semantic and exact-match needs.

Use failure cases to tune the retrieval policy

Hybrid retrieval is most valuable when its failure modes are visible. Exact model numbers, error codes, and legal clause references often reveal why sparse retrieval must remain available. Broad questions, paraphrases, and terminology mismatches reveal why dense retrieval matters. Build a small evaluation set containing both categories before changing candidate depth or fusion settings.

Operational traces should make it possible to answer a simple question for every poor result: was the relevant document absent from dense candidates, absent from BM25 candidates, or ranked poorly after fusion? That distinction separates indexing and query-representation problems from ranking-policy problems.

  • Include acronym, identifier, paraphrase, and multi-topic queries in evaluation sets.
  • Inspect zero-overlap queries, where dense and sparse paths return entirely different documents.
  • Track which retrieval path contributed each final result.
  • Revisit chunking and metadata filters when relevant documents fail to enter either candidate set.