Start with a small, representative relevance set

Create a compact evaluation set before changing embeddings, analyzers, or ranking logic. It does not need to be large to be useful: a few dozen queries can reveal recurring failure patterns when they represent real user language and real document types.

For each query, record one or more documents that should be retrieved near the top. Include queries with exact identifiers, product names, abbreviations, natural-language questions, and ambiguous terms. These categories exercise lexical and semantic retrieval differently.

  • Capture the query exactly as a user would write it.
  • Assign expected documents or document IDs, including acceptable alternatives where relevant.
  • Label the query type: identifier, phrase, question, acronym, or broad topic.
  • Keep the set versioned so changes can be compared over time.

Measure dense and sparse retrieval separately

Run every evaluation query through dense retrieval and through BM25 retrieval independently. Record whether an expected document appears in each candidate list and at what rank. This isolates which retrieval signal is carrying the query.

Dense retrieval is often most informative for paraphrases and concept-level questions, while BM25 is especially useful when exact tokens matter. These are tendencies rather than guarantees, which is why query-level evidence is more valuable than assumptions.

  • Track recall at a fixed candidate depth, such as whether an expected document appears in the first N results.
  • Track reciprocal rank to distinguish a result found at rank 1 from one found near the cutoff.
  • Group results by query type rather than relying only on an overall average.
  • Save the returned text, identifiers, and scores needed to inspect unexpected matches.

Fix the failure class, then revisit combination logic

If BM25 misses identifier-heavy queries, inspect tokenization, field selection, and whether important terms were preserved during ingestion. If dense retrieval misses clear paraphrases, inspect document chunk boundaries, the text included in each embedding, and whether the query and document content describe the same unit of meaning.

Only after each retrieval path is understandable should you adjust how their result sets are combined. A combination rule cannot reliably compensate for a dense index that never retrieves the relevant chunk or a sparse index that lost a critical identifier during document preparation.

  • Check ingestion text before changing ranking parameters.
  • Use failure examples to decide whether a document should be split, merged, or enriched with context.
  • Preserve exact names, codes, and titles in searchable fields.
  • Re-run the same evaluation set after every meaningful change and compare per-query outcomes.