Make one canonical ID the join key

Assign every retrievable unit a canonical ID before it enters either retrieval path. For a chunked knowledge base, that usually means a chunk ID rather than only a parent document ID. The same canonical ID should be written with the dense vector record and included in the sparse document indexed for BM25.

A shared ID makes hybrid retrieval a data-joining problem instead of a guessing problem. If a dense result and a BM25 result carry the same ID, they describe the same retrieval unit even when the systems return different ranking scores or different result counts.

  • Use immutable IDs; do not derive identity from a mutable title or file path.
  • Keep a parent_document_id alongside the chunk ID for grouping and display.
  • Store a content_version or source revision to distinguish replacements from older content.
  • Normalize IDs consistently across ingestion workers, query services, and downstream analytics.

Mirror retrieval-critical metadata, not every field

Dense and sparse indexes do not need identical payloads, but they should share the fields required to make the same retrieval decision. Examples include tenant or workspace identity, access scope, language, document type, publication state, and content version.

This prevents a common hybrid failure mode: one path retrieves an item that the other path would have excluded. If filtering is part of the product’s retrieval contract, define the filter values once during ingestion and apply equivalent constraints to both dense and sparse searches.

  • Treat tenant and authorization boundaries as retrieval-critical metadata.
  • Use enumerated values for fields such as language and document type where possible.
  • Define whether filtering occurs before retrieval, after retrieval, or in both stages.
  • Keep display-only fields, such as long excerpts, outside the minimal retrieval contract when they are not needed for ranking or filtering.

Version writes so updates do not create split-brain results

Documents change, and hybrid systems must account for the interval in which one index has received an update while the other has not. A version field gives the query layer and operational tooling a way to identify which revision a result represents.

For replacements, use an explicit lifecycle: write the new revision, verify that its canonical ID and metadata are valid in both retrieval paths, then retire the previous revision according to your retention policy. The exact mechanics depend on your ingestion design, but the goal is consistent visibility rather than simultaneous write timing.

  • Record ingestion status separately for dense and sparse indexing.
  • Make indexing operations idempotent so retries do not create ambiguous duplicates.
  • Log the canonical ID, parent ID, version, and source checksum for each ingestion event.
  • Test updates, deletions, and permission changes—not only first-time document ingestion.