Treat a chunk ID as the contract between retrieval paths

A document is usually split into chunks before indexing. If the dense and sparse pipelines generate their own IDs independently, the same passage can become difficult to recognize after retrieval. One system may return a vector-oriented identifier while the other returns a document-and-offset identifier, even though both point to identical text.

Instead, assign a canonical chunk ID before sending content to either index. Derive it from stable source information, such as a source document identifier, a document version, and a deterministic chunk ordinal or character range. Store that same ID with the chunk in both the S3 Vectors dense path and the Quickwit BM25 sparse path.

  • Use source IDs that survive re-indexing, not randomly generated IDs per ingestion run.
  • Include a document version or content revision in the identity when old and new versions must coexist.
  • Make chunk boundaries deterministic so a re-run produces the same IDs for unchanged content.

Deduplicate candidates before judging retrieval quality

Dense and sparse search may both retrieve the same chunk. That is useful evidence: the passage is semantically related and contains matching lexical terms. But showing it twice wastes result slots and can make a result set appear stronger than it is.

Merge candidate lists by canonical chunk ID before presentation or downstream ranking. Keep the retrieval provenance for each merged candidate—dense, sparse, or both—rather than discarding it. Provenance makes it possible to inspect why a chunk appeared without requiring dense and sparse scores to be directly comparable.

  • Union candidates by chunk ID, not by title or text similarity.
  • Record whether each candidate came from dense search, BM25, or both.
  • Preserve each path’s original rank and score as diagnostic fields.
  • Apply any final ordering only after duplicates have been collapsed.

Use provenance to tune queries and chunking

Once every result has a shared identity and source labels, retrieval failures become easier to classify. A relevant chunk found only by BM25 may indicate that exact terminology, codes, or names matter. A chunk found only by dense search may indicate vocabulary mismatch between the user’s wording and the source text. A relevant chunk found by neither path points to an indexing, chunking, or query-construction issue.

This creates a disciplined evaluation loop. Build a small set of representative queries, inspect the unique candidates from each path, and record which relevant chunks were missed or duplicated. Change one variable at a time—chunk boundaries, sparse query terms, document normalization, or candidate depth—then compare the provenance patterns again.

  • Include acronym-heavy, natural-language, and identifier-based queries in evaluation sets.
  • Inspect missed relevant chunks before changing score-combination logic.
  • Watch for duplicate chunks from overlapping windows, not just dense-versus-sparse duplicates.
  • Keep query examples and expected source chunks versioned alongside the corpus.