Start with complementary candidate sets

Dense retrieval and BM25 are sensitive to different signals. Dense search can connect differently worded text with similar meaning, while BM25 is often valuable when a query contains exact terminology that should not be paraphrased away.

A practical retrieval pipeline can issue the same query to regional S3 Vectors for dense candidates and to Quickwit BM25 for sparse candidates. Keep the retrieval stages separate initially: each system should return document identifiers, ranks, and any metadata needed for filtering or later inspection.

  • Use dense retrieval for semantic phrasing and concept-level matches.
  • Use BM25 for product names, error codes, acronyms, IDs, and uncommon vocabulary.
  • Apply the same access-control and corpus filters to both candidate paths.
  • Store a stable document or chunk ID so results can be merged safely.

Fuse ranks instead of comparing raw scores

Raw scores from dense and sparse retrieval are usually not directly comparable. Their scales, distributions, and meanings can differ, even when both are returned as numeric values. Treating one score as inherently larger can quietly bias the result list toward one retriever.

A rank-based method such as reciprocal rank fusion (RRF) avoids that assumption. For each document, add a contribution based on its position in each result list, for example 1 divided by k plus rank. The constant k reduces the impact of small rank changes near the top of a list. Documents found by both retrievers receive contributions from both paths.

  • Deduplicate by canonical document or chunk ID before presenting results.
  • Record each source rank alongside the fused score for debugging.
  • Choose a fixed candidate depth for each retriever before tuning fusion.
  • Evaluate changes with representative queries rather than relying on score magnitude.

Measure failure modes, then tune the pipeline

The best hybrid configuration depends on the queries your users actually issue. Build a small evaluation set that includes natural-language questions, exact-name lookups, acronym-heavy queries, and queries that mix a concept with a specific constraint. Label which result is useful, not merely which result contains overlapping words.

When a query fails, inspect the two candidate lists before changing embeddings, tokenization, or fusion settings. If the right item appears in neither list, the issue may be chunking, indexing coverage, or filtering. If it appears in one list but is buried after fusion, the issue is ranking. This distinction keeps tuning targeted.

  • Review dense-only, BM25-only, and fused results for each test query.
  • Track zero-result and no-useful-result cases separately.
  • Check whether chunks preserve titles, identifiers, and local context.
  • Version query handling and evaluation sets so ranking changes remain explainable.