Define a chunk as the unit of retrieval

A source document is not necessarily the right unit to index. Long pages, manuals, and support articles frequently contain multiple topics, while user questions usually target a specific passage. Split source content into chunks that can stand on their own as retrieval results.

The important design choice is to create one canonical record for every chunk before sending data to either retrieval system. Both the dense representation stored in S3 Vectors and the text indexed by Quickwit BM25 should originate from that same record.

  • Use a stable chunk_id that does not depend on index-specific internal identifiers.
  • Store the source document ID, chunk order, title, body text, and relevant metadata together.
  • Keep the text used for embedding and the text used for BM25 intentionally defined.
  • Record a content version or hash to make updates traceable.

Keep the dense and sparse views compatible

Dense search and BM25 use different signals. Dense retrieval depends on semantic similarity, while BM25 emphasizes terms found in the query and indexed text. That difference is useful only when the results can be compared and merged at the same chunk level.

Avoid maintaining separate chunking rules for the two paths unless there is a clear reason to do so. If a dense result represents an entire section but a sparse result represents one paragraph, deduplication and ranking become harder because the records overlap without being equivalent.

  • Normalize whitespace, markup handling, and document titles before producing either index input.
  • Apply the same visibility and deletion rules to both indexing paths.
  • Attach shared metadata such as tenant, language, content type, or access scope where applicable.
  • Treat re-chunking as a coordinated reindexing event rather than a change to one path only.

Merge by stable ID, then inspect disagreements

After issuing a dense query and a BM25 query, use chunk_id as the join key for result handling. A chunk returned by both paths is one record with two pieces of retrieval evidence, not two separate passages to show a user.

This also creates a useful debugging workflow. When the paths disagree, inspect the canonical chunk record: its wording, boundaries, metadata, and version. The problem may be a query formulation issue, but it may also be stale indexing or inconsistent source preparation.

  • Deduplicate candidate results by chunk_id before presenting or reranking them.
  • Preserve each path's rank and score separately for analysis.
  • Log the canonical record version alongside retrieval results.
  • Test updates, deletes, and permission changes across both dense and sparse indexes.