Why raw dense and BM25 scores should not be added directly

A dense retriever produces a similarity or distance-derived value based on an embedding space. BM25 produces a lexical relevance score based on term frequency, document frequency, and length normalization. Even when both are sorted so that higher means more relevant, the magnitude and distribution of their scores describe different things.

Adding those values with fixed weights can work for a narrow evaluation set, but it creates a calibration problem. A change to embedding model, document chunking, analyzer settings, or corpus composition can shift score distributions and invalidate earlier weights. The result may be a hybrid system that appears stable in development but behaves inconsistently as content evolves.

  • Dense search helps with paraphrases and semantic relatedness.
  • BM25 helps preserve exact-token matches such as product names and codes.
  • A score from one retriever is not automatically meaningful on the other retriever's scale.

Use reciprocal rank fusion to merge result lists

RRF assigns each document a contribution based on its position in each ranked list, rather than its raw score. For a document d, compute RRF(d) = Σ 1 / (k + rank_i(d)), where rank_i(d) is the one-based rank from retriever i and k is a positive constant. Documents absent from a list contribute nothing from that retriever.

For Talqora applications, issue a dense query against the regional S3 Vectors-backed path and a sparse query against the Quickwit BM25-backed path. Retrieve a moderate candidate list from each, deduplicate by a stable document or chunk identifier, calculate RRF in the application layer, then return the highest fused ranks. A commonly used starting value for k is 60, but it should be treated as a tunable parameter rather than a universal default.

  • Request the same candidate depth from both retrievers initially, such as the top 50 results.
  • Keep the original dense rank and BM25 rank for debugging and evaluation.
  • Deduplicate before final ranking so a shared chunk receives contributions from both lists.
  • Apply filters consistently to both retrieval paths before fusion.

Evaluate the failure cases, not only average relevance

RRF is deliberately simple, but it still needs evaluation against the queries that matter to the application. Build a small labeled set containing exact-lookup queries, natural-language questions, ambiguous short queries, and terminology that appears in multiple documents. Compare dense-only, BM25-only, and fused rankings using the same filters and corpus snapshot.

Inspect disagreements manually. If a document ranks highly because it appears in both lists, that is often a useful relevance signal. If either retriever contributes recurring noise, adjust its candidate depth, apply query routing for clearly lexical requests, or improve the source content and metadata. Keep fusion logic observable: logging component ranks makes it far easier to explain why an item was returned.

  • Track whether the intended document appears in the first few results for each query class.
  • Test identifier-heavy queries separately from conceptual questions.
  • Re-evaluate after changing embeddings, chunk boundaries, BM25 analysis, or filters.
  • Preserve component ranks in logs to support debugging and relevance review.