Treat a chunk as a durable retrieval unit

A source document is rarely the right unit for retrieval. Long documents commonly need to be split into smaller chunks so that an embedding represents a focused passage and a lexical index can match the terms near an answer. The important design decision is to define exactly what makes one chunk distinct.

A useful chunk record includes the source document identity, a chunk position or section identity, the chunk text, and metadata needed by the application. Both the dense and sparse indexing paths should receive the same canonical chunk record, rather than independently rebuilding chunk boundaries from the original source.

  • Use one canonical source of chunk text for every index.
  • Store the parent document ID alongside the chunk ID.
  • Keep chunk order or offsets when result context matters.
  • Include only metadata that has a clear retrieval or filtering purpose.

Build IDs from deterministic inputs

A deterministic ID is generated from values that are stable for the same logical chunk. A common pattern is a namespaced source identifier plus a chunk ordinal, such as `handbook:leave-policy:0007`. Another option is to include a content hash when the ingestion system needs to recognize whether the text has changed.

Avoid random IDs for chunks that may be reprocessed. Random identifiers make a repeat ingestion look like new content, which can leave older representations behind unless every prior record is explicitly found and removed. Deterministic IDs make retries and rebuilds easier to make idempotent.

  • Namespace IDs by tenant, corpus, or source system when appropriate.
  • Use a fixed-width ordinal if chunk positions are part of the identity.
  • Hash normalized text only after defining normalization rules.
  • Version the ID scheme if a chunking-policy change intentionally creates a new layout.

Plan updates around alignment, not just ingestion

When a document changes, compare its newly generated chunk set with the previously indexed set. Chunks with unchanged IDs and unchanged content can be left alone when the surrounding system supports that workflow. Changed chunks should be re-indexed consistently in both the dense and sparse representations, while removed chunk IDs should be deleted from both.

At query time, stable IDs also simplify result handling. A dense result and a BM25 result that point to the same chunk ID can be recognized as the same passage before application-level ranking, deduplication, or context assembly. The ID becomes the join key between retrieval signals and the metadata needed to present a useful answer.

  • Record the chunking-policy version with each ingestion run.
  • Maintain a manifest of chunk IDs produced for each source document.
  • Delete obsolete chunks from every representation during updates.
  • Use the chunk ID as the application-level key for deduplication and result assembly.