Define the chunk as the unit of retrieval

A document is usually too large and too mutable to serve as the direct unit of retrieval. Instead, split it into chunks that can be independently embedded, indexed for BM25, displayed to users, and refreshed when source content changes.

Give every chunk a canonical identifier derived from durable source attributes rather than its position in a single indexing run. A useful ID might combine a source-system name, a document identifier, a content version or revision, and a stable chunk key. The exact format matters less than its ability to be reproduced deterministically.

  • Use one chunk ID in both the dense and sparse indexes.
  • Keep the source document ID separately for grouping and filtering.
  • Store a content fingerprint to detect whether a chunk’s text changed.
  • Avoid IDs based only on ingestion timestamps or random UUIDs.

Keep retrieval metadata aligned across both indexes

A matching ID is necessary, but it is not sufficient. Each retrieval path needs enough shared metadata to support the same operational decisions: tenant boundaries, document type, language, permissions, source revision, and deletion state are common examples.

Treat this metadata as an indexing contract. When a field is added, renamed, or changes meaning, update the dense and sparse ingestion paths together. Otherwise, a filter applied to one path can silently produce a different eligible corpus than the other.

  • Maintain a versioned schema for chunk metadata.
  • Apply tenant and authorization filters before presenting retrieved content.
  • Record the embedding model or embedding revision alongside dense entries.
  • Use explicit soft-delete or active-state fields during asynchronous cleanup.

Make updates idempotent and make joins observable

Content updates may arrive more than once, and dense and sparse indexing may complete at different times. Design ingestion so that writing the same chunk revision again has the same final state. This reduces duplicate results and makes retry behavior safer when a job is interrupted.

At query time, preserve the retrieval origin for every candidate: dense, BM25, or both. Joining candidates by canonical chunk ID makes it possible to inspect overlap, identify missing entries, and diagnose cases where one index has a newer revision than the other. These diagnostics are often more useful than tuning ranking weights prematurely.

  • Log chunk ID, source revision, retrieval path, and query request ID.
  • Track candidates found in only one retrieval path.
  • Reject or quarantine writes with missing required metadata.
  • Test inserts, updates, deletes, and retries against both indexes as one workflow.