Treat a chunk as a first-class record

A source document is rarely the right retrieval unit. Long pages, manuals, tickets, and knowledge-base articles commonly need to be split into smaller passages so that both semantic and lexical retrieval can return focused evidence. Once content is chunked, each chunk should be treated as its own durable record rather than as an anonymous position in an ingestion batch.

Create an identifier from fields that describe the chunk's identity, not from the embedding itself. A useful identity usually includes a stable source-document ID, a content version or revision, and a deterministic chunk position. For example, a chunk might be represented by a source key plus a zero-based ordinal, while its text hash is stored separately for change detection.

  • Use a stable source-document ID that survives title or path changes where possible.
  • Assign chunk ordinals after applying a deterministic chunking procedure.
  • Store the document revision alongside the chunk ID rather than overwriting provenance.
  • Keep a content hash to detect whether a chunk's text changed.

Send the same identity to dense and sparse ingestion

Dense search and BM25 search index different representations of a passage, but they should point to the same retrievable unit. During ingestion, emit one chunk record, then derive the dense payload and sparse payload from that record. The dense path receives the chunk's vector representation; the sparse path receives text appropriate for BM25 indexing. Both retain the canonical chunk ID.

This alignment is especially helpful when an application gathers candidates from separate retrieval paths. A result can be recognized as the same passage even when its rank differs between dense and sparse search. The application can then merge, inspect, or present results without relying on fragile text equality.

  • Generate IDs before creating embeddings or building sparse-search documents.
  • Store source reference, title, chunk ordinal, revision, and text hash with each chunk record.
  • Avoid using array offsets from a transient batch as persistent IDs.
  • Do not derive a chunk ID from normalized text alone; identical boilerplate can occur in multiple sources.

Make updates and debugging reproducible

Canonical IDs also make lifecycle handling clearer. When a document changes, compare newly generated chunk hashes with the prior revision. Unchanged chunks can retain their lineage, while changed or removed chunks are explicit events in the ingestion process. This reduces ambiguity about whether a returned passage came from the current source revision or an older representation.

For retrieval debugging, return or log the canonical chunk ID with the retrieval result. An engineer can use that ID to locate the original text, inspect its source revision, verify chunk boundaries, and compare how the passage was represented for dense and BM25 retrieval. This turns relevance investigations into a record-level workflow instead of a search for similar-looking text.

  • Record the chunking configuration and embedding-model version as ingestion metadata.
  • Keep source revision information available to downstream result rendering.
  • Log retrieval path, rank, score when available, and canonical chunk ID together.
  • Test ingestion by confirming that every intended chunk has matching dense and sparse identities.