Treat identity as retrieval infrastructure

Dense search typically works from embeddings created for a chunk of text, while BM25 works from indexed terms in a text field. A single source document can therefore produce multiple records: several chunks for dense retrieval and one or more searchable fields for sparse retrieval. Those records need a stable way to refer back to the same source content.

Use an application-controlled canonical ID rather than relying on an identifier generated by either retrieval system. A useful ID identifies the tenant or corpus, the source document, the content unit, and the revision. For example, a chunk identity might be derived from a tenant ID, a source document ID, a chunk ordinal, and a content version.

  • Keep the canonical ID independent of embedding model choice.
  • Include tenant or corpus scope in the identity when data is multi-tenant.
  • Store a source-document ID alongside chunk-level IDs.
  • Make revisions explicit instead of silently overwriting content.

Build one ingestion manifest before writing to either index

Create a manifest from the source document before generating embeddings or preparing BM25 fields. Each manifest row should describe one retrievable unit and contain its canonical ID, source ID, revision, text, metadata, and deletion state. The dense and sparse indexing jobs can then consume the same manifest rather than independently reconstructing document structure.

This pattern reduces subtle mismatches. If chunk boundaries change in the embedding pipeline but the sparse pipeline still indexes an older document shape, a result may be difficult to resolve or present. A shared manifest makes the intended relationship visible and gives ingestion jobs a common input to validate.

  • Generate chunk boundaries once and record them in the manifest.
  • Record the content hash used for a revision.
  • Carry display metadata, such as title and source URI, with the retrievable unit.
  • Persist an ingestion status for each dense and sparse write.

Make updates and debugging version-aware

Updates are where identity discipline becomes operationally valuable. When a source changes, create a new manifest revision, write its dense and sparse representations, and only then make that revision eligible for retrieval in your application. Retain enough state to remove or exclude the prior revision according to the retention rules of the corpus.

The same fields also improve incident investigation. When a result looks stale, compare the canonical ID, revision, content hash, and indexing status across the dense and sparse records. This is more reliable than comparing result text by eye, especially when documents have repeated passages or similar titles.

  • Log canonical ID and revision with every retrieval result.
  • Use revision filters or application-side eligibility checks to avoid serving superseded content.
  • Monitor for manifest rows that were written to only one retrieval path.
  • Test delete and re-index flows with the same care as initial ingestion.