Why raw score blending is fragile

It is tempting to add a dense-search score to a BM25 score and sort by the total. That approach assumes both scores share a stable meaning and range. In practice, dense similarity depends on the embedding model, vector normalization, and similarity function, while BM25 depends on corpus statistics and term-frequency behavior.

Even if each retriever works well independently, a fixed weighted sum can behave differently after a corpus grows, an embedding model changes, or the distribution of queries shifts. A score threshold that looked sensible for one retriever is not automatically comparable to a threshold from the other.

  • Dense retrieval emphasizes semantic proximity.
  • BM25 emphasizes lexical overlap and term rarity.
  • Score ranges can change independently over time.
  • Rank positions are often more comparable than raw scores.

Fuse ranked lists with reciprocal rank fusion

Reciprocal rank fusion (RRF) combines result lists using rank rather than raw score. For each document returned by a retriever, add a contribution of 1 divided by k plus its rank. A document that appears near the top of either list receives a meaningful boost; a document found by both receives contributions from both lists.

For a query, retrieve a candidate list from S3 Vectors and a candidate list from Quickwit BM25. Deduplicate by a stable document or chunk identifier, calculate the RRF total for every candidate, then sort descending. The constant k reduces the difference between adjacent top ranks and prevents rank one from overwhelming the rest of the list.

  • RRF score: sum of 1 / (k + rank) across retrieval lists.
  • Use one-based ranks: first result has rank 1.
  • Choose a consistent candidate depth for both retrievers.
  • Keep document IDs stable so duplicates can be merged correctly.

Operate the fusion layer as a retrievable system

Start with a simple, inspectable pipeline: log the query, the dense rank, the BM25 rank, the fused rank, and the selected document ID. These fields make it possible to explain why an item was returned and to identify patterns such as identifier-heavy queries that rely on BM25 or paraphrased questions that rely on dense retrieval.

Evaluate on a small set of representative queries before changing candidate depth or k. Include exact-name queries, error codes, short ambiguous queries, natural-language questions, and queries containing vocabulary that may not have appeared frequently in the corpus. Review relevance at the final display depth, not only whether a relevant item entered either candidate list.

  • Log per-retriever ranks alongside fused rank.
  • Test exact-match and semantic-query cases separately.
  • Treat k and candidate depth as evaluation parameters.
  • Re-evaluate after corpus, embedding, or indexing changes.