Why raw-score blending is fragile
Dense retrieval ranks documents by semantic proximity between embeddings. BM25 ranks documents from term statistics and query-document token overlap. Both produce ordered results, but their numeric scores arise from different calculations and should not automatically be treated as comparable.
A weighted formula such as dense_score + bm25_score can work only after careful normalization and evaluation. Score distributions may also vary by query: a highly specific keyword query and a broad conceptual query can produce very different ranges. This makes one global weighting rule difficult to reason about.
- Dense search can recover paraphrases and conceptually related passages.
- BM25 can preserve exact terms, identifiers, and uncommon vocabulary.
- Rank position is usually easier to combine across retrieval methods than raw score magnitude.
Fuse two candidate lists with RRF
Run the same user query through dense retrieval and BM25 retrieval, then collect a bounded candidate list from each. For every document, add a contribution based on its position in each list: 1 divided by k plus the document rank. Documents returned by both methods accumulate contributions, while a strong result from only one method can still remain competitive.
The constant k reduces the difference between nearby ranks and prevents the first position from overwhelming the rest of the list. Treat k and each retriever's candidate depth as evaluation parameters, not universal defaults. The goal is to produce one fused candidate set that can be returned directly or passed to a later reranking stage.
- Deduplicate by a stable document or chunk identifier before emitting results.
- Use one-based ranks consistently in every result list.
- Assign no contribution when a document is absent from a retriever's list.
- Keep the original dense and BM25 ranks in diagnostics for debugging.
Make fusion observable before tuning it
Start with a small, representative query set that includes natural-language questions, exact-name lookups, technical terms, and ambiguous requests. Inspect whether relevant documents appear in the dense list, the BM25 list, both, or neither. This tells you whether a poor fused result is a fusion issue or a candidate-generation issue.
Log the fused rank together with each source rank and the source lists that contributed to it. These records make it possible to explain why a result moved up, detect duplicate chunks, and decide whether one retriever needs a deeper candidate list. Tune against relevance judgments or review examples rather than relying on score intuition alone.
- Evaluate dense-only, BM25-only, and fused rankings on the same queries.
- Include queries with acronyms, product names, and exact error text.
- Review documents that appear in only one retrieval path.
- Revisit candidate depth when relevant documents never reach the fusion step.