Why raw-score blending is fragile

Dense retrieval ranks documents by the relationship between a query embedding and document embeddings. BM25 ranks documents using term statistics such as matching terms, document frequency, and field length. Their resulting numbers have different meanings and can vary substantially across queries.

Adding a dense score to a BM25 score, even with a fixed weight, assumes both score distributions are stable and comparable. That assumption can fail when a query contains a rare product code, a short ambiguous phrase, or a broad conceptual request. A score that looks large in one retrieval system is not automatically stronger evidence than a smaller score in the other.

  • Use raw scores for diagnostics, not as an automatic shared relevance scale.
  • Inspect result overlap: documents returned by both methods are often useful signals.
  • Keep dense and sparse retrieval independently observable during tuning.

Fuse ranked lists with reciprocal rank fusion

RRF combines rankings rather than score values. Run dense search against S3 Vectors and BM25 search through Quickwit, retain a candidate list from each, then assign every document a fused score based on its position in each list. A document that appears in both lists receives contributions from both ranks.

A common formulation is RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based position of document d in result list i. The constant k reduces the difference between adjacent top positions and prevents the first few ranks from overwhelming every other contribution. Choose k deliberately, document it, and evaluate it against representative queries rather than assuming one value is universal.

  • Fetch a sufficiently deep candidate list from each retriever before fusion.
  • Deduplicate by a stable document identifier before calculating final order.
  • Assign no contribution for a document absent from a particular result list.
  • Apply a deterministic tie-breaker, such as document ID, for reproducible responses.

Treat fusion as a retrievable, testable layer

Put fusion in application logic or a dedicated retrieval layer so its inputs and outputs can be recorded. For each query, capture the dense rank, BM25 rank, fused rank, and the document identifiers considered. This makes it possible to explain why a result moved up or down after a change.

Build a small evaluation set that reflects actual retrieval work: exact identifiers, vocabulary mismatch, multi-term research questions, and queries with expected empty results. Review failures by retrieval source. If BM25 misses identifiers, investigate indexing and tokenization; if dense retrieval misses paraphrases, inspect document chunking and embedding choices before changing fusion rules.

  • Version fusion parameters alongside query-processing changes.
  • Compare dense-only, BM25-only, and fused rankings on the same query set.
  • Monitor empty-result rates and overlap between candidate lists.
  • Use judgment labels or structured review notes to guide changes.