Why raw-score merging is fragile

Dense retrieval and BM25 produce scores with different meanings. A vector similarity score reflects the distance or similarity behavior of a chosen embedding model and index configuration. A BM25 score reflects term frequency, document frequency, field length, and query terms. Even when both are returned as numbers, their scales are not naturally comparable.

A common mistake is to normalize each score list and add the values together. That can work in a tightly controlled evaluation setup, but it can also become sensitive to query length, corpus growth, model changes, and the number of candidates returned by each retriever. Rank-based fusion avoids assuming that a score of 0.8 from one system has the same significance as a score of 0.8 from another.

  • Dense search helps with paraphrases, concepts, and vocabulary mismatch.
  • BM25 helps with identifiers, rare names, literals, and exact phrases.
  • Raw score ranges can shift as models, analyzers, or corpora change.

Fuse candidate lists with RRF

RRF assigns each document a contribution based on its rank in each result list. For a document d, calculate RRF(d) = sum of 1 divided by (k + rank_i(d)) across the lists where d appears. The constant k reduces the difference between nearby top ranks and prevents the first position from dominating the combined result too aggressively.

For each query, request a candidate set from S3 Vectors and a candidate set from Quickwit BM25. Deduplicate by a stable document or chunk identifier, calculate the fused score, sort descending, and return the top results. A document found by both retrievers receives two contributions, while a strong result unique to either retriever can still surface.

  • Use the same chunk identifier in dense and sparse indexes.
  • Keep rank positions one-based when implementing the formula.
  • Start with a fixed candidate depth for both retrievers so behavior is easier to inspect.
  • Treat k as an evaluation parameter rather than an assumed universal constant.

Evaluate failures before adding complexity

Build a small query set from realistic user tasks, including acronym lookups, exact product codes, conceptual questions, and queries that mix a precise term with a broad intent. For every query, inspect not only whether a relevant chunk appears, but also which retriever supplied it and how fusion changed its final position.

RRF is intentionally simple, which makes it a useful baseline. If it repeatedly over-ranks lexical matches for semantic questions, or under-ranks exact matches for identifier-heavy questions, investigate the candidate depths, chunking, metadata filters, and query construction first. More elaborate learned reranking or score calibration is easier to justify after a rank-fusion baseline exposes a specific failure mode.

  • Record dense rank, BM25 rank, fused rank, and relevance judgment for sampled queries.
  • Test documents returned by only one retriever as well as documents returned by both.
  • Separate retrieval quality issues from chunking and metadata-filtering issues.
  • Re-evaluate after embedding, analyzer, or corpus changes.