Treat a chunk as a versioned record, not an anonymous embedding

A chunk should have an identity that survives retries and makes updates explicit. Random IDs are convenient at ingestion time, but they make it harder to determine whether a later request is creating a new passage, replacing an existing one, or replaying an earlier write.

A practical approach is to build a deterministic ID from a stable document identifier, a chunking policy identifier, and a chunk locator. The locator can be a section path, a source offset range, or another durable position within the source. Store the source revision separately so that the current representation and its provenance are both visible.

  • Use a stable document ID that does not change when a title or URL formatting changes.
  • Include the chunking policy or schema version in the identity or metadata.
  • Keep source revision, content hash, and ingestion timestamp as metadata.
  • Avoid deriving identity solely from embedding model output.

Make dense and sparse records share the same retrieval key

Talqora uses regional S3 Vectors for dense search and Quickwit BM25 for sparse search. Regardless of how results are later combined, both representations should point to the same canonical chunk record. A shared chunk ID lets an application recognize that a dense hit and a BM25 hit refer to the same passage.

The shared key is also useful outside ranking. It gives downstream code one handle for fetching display text, applying permissions, recording feedback, and removing stale entries. Put searchable text and filtering metadata under a consistent schema, then write the dense and sparse representations from the same prepared chunk payload.

  • Generate one canonical chunk ID before writing to either retrieval system.
  • Persist the original chunk text or a durable reference to it alongside the ID.
  • Use identical tenant, document, and access-control metadata conventions across representations.
  • Deduplicate retrieved results by canonical chunk ID before presenting them.

Update by replacement, then verify for drift

When source content changes, first identify the affected document and its prior chunks. Write the replacement chunks using the new revision, then remove records that belonged only to the prior revision. This avoids leaving passages from deleted sections available to retrieval after an update.

Verification should check more than whether an ingestion request succeeded. Compare expected chunk IDs with the IDs created for the new revision, confirm that obsolete IDs are absent, and inspect a small set of changed documents through the retrieval path. These checks catch chunk-boundary changes, failed deletes, and metadata mismatches that can otherwise look like ranking problems.

  • Maintain a document-to-chunk manifest for each source revision.
  • Make ingestion operations safe to retry with deterministic IDs.
  • Track write and delete outcomes separately for dense and sparse representations.
  • Run periodic reconciliation between the source manifest and indexed chunk metadata.