Treat a chunk ID as a retrieval contract

A chunk ID should identify a logical retrieval unit, not an incidental processing event. Random IDs created on every ingestion run make it difficult to tell whether a result represents a new chunk, an updated version of an existing chunk, or a duplicate created by reprocessing.

Use an ID assembled from durable source attributes: a source namespace, a canonical document ID, a content version or revision, and a chunk position. The same identifier can accompany the chunk sent to dense and sparse retrieval paths, giving downstream code a common key for joining or comparing results.

  • Prefer canonical document IDs over filenames, which may change after an export or rename.
  • Include a source namespace when multiple repositories can contain the same document ID.
  • Use a deterministic chunk ordinal only when the chunking procedure is itself stable.
  • Store the original document ID separately so application code can group results by document.

Version content without confusing old and new chunks

Document updates introduce a choice: overwrite the prior retrieval representation or retain multiple versions temporarily. A version-aware ID makes that choice explicit. For example, a logical shape such as source:document:revision:chunk can distinguish chunks from two revisions even when their text and position are similar.

The revision value should reflect the source of truth whenever possible, such as a repository commit, a content-management revision, or a normalized content hash. Avoid using ingestion time as the only version marker: it describes when processing happened, not necessarily whether the source changed.

  • Use a source-provided revision when one exists.
  • Use a normalized-content hash when the source has no reliable revision field.
  • Record ingestion time as metadata for operations and debugging, not as the document version itself.
  • Define a clear policy for which revision is eligible for retrieval before indexing new content.

Make hybrid result merging deterministic

Dense search over S3 Vectors and sparse search with Quickwit BM25 may return overlapping as well as distinct candidates. Stable IDs allow the application to recognize overlap before applying any ranking or fusion policy. Without them, one chunk can appear twice simply because the dense and sparse pipelines assigned unrelated identifiers.

Keep the retrieval payload sufficient for validation: the stable chunk ID, document ID, revision, chunk ordinal, and any access-control or tenant fields needed by the application. When a dense and sparse result share a chunk ID, they refer to the same retrieval unit; when they share only a document ID, they may be different passages from the same document.

  • Deduplicate exact chunk matches by stable chunk ID before presenting results.
  • Preserve both dense and sparse scores if a later fusion step needs them.
  • Apply tenant and access-control constraints consistently to every retrieval path.
  • Log the stable ID and revision with selected results to make relevance investigations reproducible.