Start with query intent, not a ranking formula

Dense and sparse retrieval respond to different kinds of evidence. Dense retrieval is useful when a query and a document express similar meaning with different wording. BM25 is useful when exact terms, identifiers, product names, error codes, and other lexical details matter.

A useful first step is to label a small, representative set of production-like queries by intent. The goal is not to create a perfect taxonomy; it is to make disagreements between dense and sparse results explainable. A query such as an error message may depend on exact tokens, while a natural-language how-to question may tolerate vocabulary variation.

Keep this evaluation set versioned. Retrieval changes are difficult to assess when the query sample quietly changes at the same time.

  • Exact-match queries: IDs, SKUs, error strings, filenames, quoted phrases
  • Conceptual queries: questions, paraphrases, task descriptions
  • Mixed queries: a concept plus a required product name, version, or constraint
  • Ambiguous queries: short terms with multiple possible meanings

Inspect the result sets independently

For each evaluation query, retrieve a fixed-depth result list from dense search and another from BM25. Record the query, the returned document identifiers, ranks, and an editorial relevance judgment. This makes overlap and disagreement visible without assuming that either path is the default winner.

Do not evaluate only whether a relevant document appears somewhere in a long list. Its position matters. A document that is relevant but consistently buried may not help the user experience, and a top-ranked near-match can be more harmful than an obvious miss.

Talqora’s underlying retrieval paths make this separation especially practical: regional S3 Vectors provides the dense path, while Quickwit BM25 provides the sparse path. Treat their outputs as evidence to analyze before treating them as scores to combine.

  • Relevant documents found by both paths
  • Relevant documents found only by dense retrieval
  • Relevant documents found only by BM25
  • Top-ranked but irrelevant results and the terms or concepts that caused them

Use disagreement to define a blending policy

After reviewing the sample, define a simple policy that reflects observed query behavior. For example, exact identifiers and quoted error text may deserve stronger lexical treatment, while broad natural-language questions may benefit more from semantic candidates. Mixed queries often need both candidate sets because one path can preserve required terms while the other can recover paraphrases.

If you later combine or rerank candidates in application logic, normalize and validate carefully. Dense similarity values and BM25 scores are not automatically comparable simply because both are numbers. A weighting scheme should be evaluated on held-out queries and revisited whenever content, embeddings, analyzers, or query patterns change.

The durable outcome is not a universal dense-to-sparse weight. It is an evaluation loop: classify queries, inspect both paths, document failure modes, and make ranking choices that can be traced back to real retrieval evidence.

  • Keep a held-out query set for checking ranking changes
  • Log the retrieval path that supplied each candidate
  • Review zero-result and low-relevance queries regularly
  • Separate content-quality issues from retrieval and ranking issues