Why raw dense and sparse scores are difficult to merge

A dense retrieval score reflects the relationship between query and document embeddings. A BM25 score reflects term frequency, inverse document frequency, and document-length normalization. Even when both systems return larger-is-better scores, the values do not share a stable meaning or range.

A weighted formula such as `0.5 × dense_score + 0.5 × bm25_score` can therefore behave unpredictably. A small change in embedding model, index configuration, corpus composition, or BM25 analyzer may change one score distribution without changing the other. The resulting weight is often tied to a particular dataset and must be revisited as the system evolves.

  • Dense retrieval helps with paraphrases, related concepts, and descriptive queries.
  • BM25 helps preserve exact matching for names, codes, error messages, and rare terms.
  • Score magnitudes are implementation- and corpus-dependent.
  • Ranking position is usually easier to compare than raw score values.

Apply Reciprocal Rank Fusion to two candidate lists

Retrieve a bounded list of document identifiers from each source: one list from dense search in regional S3 Vectors and one from Quickwit BM25. Then calculate an RRF score for every identifier found in either list. For a document d, the score is the sum of `1 / (k + rank)` over the lists in which d appears.

The constant k reduces the difference between adjacent ranks near the top of a list. A commonly used starting point is 60, but it is a tuning parameter rather than a universal rule. Lower values emphasize top-ranked positions more strongly; higher values make rank differences less sharp. Start with a fixed value, inspect representative queries, and change it only when the observed ranking behavior supports doing so.

  • Request the same candidate depth from both retrievers as an initial baseline.
  • Use a stable document ID shared by the dense and sparse indexes.
  • Assign rank 1 to the first result in each list.
  • Sort the union of candidate IDs by descending RRF score, then use a deterministic tie-breaker.

Make fusion observable before making it complex

Log the query, the dense rank, the BM25 rank, the fused rank, and the final document ID. These fields make it possible to understand whether a result won because both methods agreed, because BM25 found an exact phrase, or because dense retrieval recovered a semantic match that sparse search missed.

Evaluate with queries that reflect real retrieval pressure: product names, abbreviations, long natural-language questions, misspellings, and mixed queries containing both an identifier and a description. If a result needs to be retrieved for filtering, permissions, freshness, or document type, apply those constraints consistently before fusion so that neither candidate list contains ineligible documents.

  • Keep the pre-fusion lists during debugging; the final list alone hides useful evidence.
  • Review failures by query class rather than relying only on aggregate relevance judgments.
  • Deduplicate by canonical document ID before presenting results.
  • Add reranking only after the fused candidate set is understood and measured.