Why raw-score mixing is fragile

A dense retriever ranks documents according to vector similarity, while BM25 ranks them from term-level matching statistics. Even when both systems return numeric scores, those values are not automatically comparable. A score of 0.7 from one ranking method does not inherently mean the same thing as a score of 0.7 from another.

Adding or averaging raw scores can therefore make ranking behavior sensitive to implementation details, score ranges, and query composition. A safer starting point is to treat each retriever as a source of ordered candidates rather than a source of universally calibrated scores.

  • Dense search can recover semantically related wording.
  • BM25 can reward exact terms, identifiers, and rare vocabulary.
  • Each method may rank the same document differently.
  • Rank position is easier to combine than unrelated score scales.

Apply reciprocal rank fusion to two result lists

RRF assigns each document a contribution based on its rank in every list where it appears. For a document d, calculate RRF(d) as the sum of 1 divided by k plus its rank, across the dense and BM25 result lists. The constant k reduces the impact of small rank differences near the top of a list.

For example, retrieve a candidate set from dense search backed by regional S3 Vectors and another from Quickwit BM25. Deduplicate documents by a stable document ID, calculate the RRF value for each candidate, then sort candidates by the fused value. A document present in both lists will usually receive a useful boost without requiring score normalization.

  • Retrieve the top N documents from dense search.
  • Retrieve the top N documents from BM25 using the same query context.
  • Use a stable ID to merge duplicate documents.
  • Compute a fused score from ranks, then return the highest-ranked merged results.

Make fusion observable before tuning it

Start with a fixed candidate depth and a documented k value, then log enough information to understand why a document was returned. Useful fields include dense rank, BM25 rank, fused rank, whether the document appeared in one or both lists, and the query class. This makes result changes explainable during relevance review.

Evaluate representative queries rather than relying only on aggregate impressions. Exact-name queries, acronym-heavy technical queries, conceptual questions, and short ambiguous queries can each reveal different retrieval behavior. If one source consistently adds little value for a query class, adjust the retrieval policy or candidate depth instead of assuming one global setting is best.

  • Record per-source ranks alongside the final fused rank.
  • Keep a small reviewed query set with expected useful documents.
  • Inspect documents that appear in both lists and documents unique to one list.
  • Change one parameter at a time when reviewing relevance.