Why raw-score blending is risky

A dense retriever typically ranks documents by embedding similarity, while BM25 ranks them using term-frequency and document-statistics signals. Even when both systems return a number called a score, those numbers represent different calculations and may have very different ranges.

Adding or averaging raw scores can therefore make one retriever dominate for mathematical reasons rather than relevance reasons. A safer initial design is to preserve each retriever’s ordering and combine the orderings after retrieval.

  • Dense search can recover semantic matches with limited term overlap.
  • BM25 can strongly reward exact names, identifiers, and rare terms.
  • Score scales may change with index configuration, corpus changes, or query shape.
  • Rank positions are easier to combine than unrelated score magnitudes.

Fuse dense and sparse rankings with RRF

For each query, retrieve a candidate list from regional S3 Vectors and a candidate list from Quickwit BM25. Assign every returned document a rank beginning at 1. For each unique document ID, sum a small contribution from every list in which it appears.

The standard RRF contribution is 1 divided by k plus the document rank. The constant k reduces the advantage of being first by a small margin and makes the fusion less sensitive to minor ranking changes near the top of either list.

  • RRF score: sum of 1 / (k + rank) across result lists.
  • Use the same canonical document ID in dense and sparse indexes.
  • Choose candidate depths deliberately; a document absent from a list contributes nothing from that retriever.
  • Start with a conventional positive k value, then validate it against representative relevance judgments.

Make fusion observable and testable

Store enough retrieval metadata to explain each fused result: dense rank, BM25 rank, per-list contribution, and final RRF score. This turns a confusing relevance report into a concrete question: did the document enter from dense search, sparse search, or both?

Evaluate with a fixed query set that includes semantic questions, exact-title requests, product codes, abbreviations, and mixed queries. Compare dense-only, BM25-only, and fused rankings. The goal is not to assume hybrid retrieval always wins, but to identify where each retrieval signal provides useful coverage.

  • Deduplicate candidates before presenting the final ranking.
  • Apply access-control and document-state filters consistently to both retrieval paths.
  • Log retrieval-list sizes and empty-result cases.
  • Inspect queries where the fused top results differ substantially from either individual list.