Treat dense and sparse search as complementary retrievers

Dense retrieval encodes meaning into vectors, making it useful when a query and a relevant document use different language. A search for “how to rotate access credentials,” for example, may still retrieve content titled “credential renewal procedure” even if the wording does not overlap exactly.

Sparse retrieval with BM25 scores term overlap and term rarity. It is especially valuable for error codes, product names, filenames, quoted text, version strings, and other tokens whose exact presence matters. Asking one retrieval method to cover every query type can create avoidable blind spots.

  • Use dense search to broaden semantic recall.
  • Use BM25 to retain precise lexical matches.
  • Keep document identifiers and metadata consistent across both indexes.
  • Return scores and source information with every candidate for debugging.

Retrieve separately before you fuse

Run the same user query through two paths: embed it for dense search against regional S3 Vectors, and submit its text to Quickwit BM25. Request a bounded candidate list from each path. The right depth depends on corpus size, query mix, and downstream ranking strategy, so it should be treated as a tunable retrieval parameter rather than a fixed constant.

Do not compare raw dense and BM25 scores directly unless their scales have been deliberately calibrated. Similar-looking numeric values can have very different meanings across retrieval systems. Instead, begin with rank-based fusion, which uses each result’s position rather than its raw score.

  • Deduplicate candidates by a stable document or chunk ID.
  • Preserve each candidate’s dense rank and sparse rank.
  • Use a timeout or fallback policy so one delayed path does not silently distort results.
  • Log candidate overlap: high overlap and zero overlap are both useful diagnostic signals.

Start with reciprocal rank fusion and evaluate by query class

Reciprocal rank fusion is a straightforward baseline: each document receives a contribution from every list in which it appears, based on the inverse of its rank offset by a constant. Documents that rank well in either list can surface, while documents supported by both lists receive an additional boost. This avoids assuming that dense and BM25 scores are numerically comparable.

Evaluate the fused results by query class, not just an overall average. Build a small judgment set containing semantic paraphrases, exact identifier searches, mixed natural-language-and-keyword queries, and short ambiguous queries. Inspect missed results and unexpected promotions to determine whether the issue is chunking, indexing, candidate depth, query construction, or fusion.

  • Use rank-based fusion as an initial, interpretable baseline.
  • Track recall-oriented measures before optimizing final ordering.
  • Test exact-token queries separately from paraphrase queries.
  • Change one retrieval parameter at a time and retain query-level evaluation notes.