Use one canonical ID across retrieval indexes

Assign every retrievable unit a canonical identifier before it is written to either index. A unit may be a product, support article, policy section, code function, or document chunk. The same canonical ID should accompany its dense-vector record and its BM25 document.

Avoid IDs derived only from insertion order or an index-specific internal key. Those values can change during rebuilds and make it difficult to join candidate lists. Instead, derive the ID from a durable source identifier plus the retrieval unit boundary.

  • Good pattern: source_document_id + section_or_chunk_id
  • Keep the ID opaque to users even if it is deterministic internally
  • Store the source document ID separately for document-level grouping
  • Use the canonical ID in application logs and retrieval traces

Make content versions explicit

A stable ID identifies the logical retrieval unit, but content can change. Add a version field or content digest to each indexed record so the application can distinguish an updated chunk from an older representation of that same chunk.

Versioning prevents a subtle failure mode: dense results may refer to newly embedded text while sparse results still describe an earlier revision. A query-time filter, or a validation step after retrieval, can ensure that only the intended version reaches ranking and presentation.

  • Record a content hash for the exact text used at indexing time
  • Include an index generation or published-at version in metadata
  • Treat text changes, chunk-boundary changes, and metadata changes as deliberate update events
  • Retain enough provenance to locate the original source and revision

Join candidates before presenting results

Dense candidates from S3 Vectors and sparse candidates from Quickwit BM25 can be represented in the application as records keyed by the canonical ID. This allows a retrieval layer to deduplicate matches, retain which path produced each candidate, and fetch the current source metadata from a trusted store.

Keep retrieval evidence alongside the candidate rather than replacing it. For example, retain the dense and sparse rank positions, the matched ID, and the content version. This makes relevance investigations concrete when a result is surprising or missing.

  • Deduplicate by canonical ID, not by displayed title or text
  • Keep per-path rank and score data as diagnostic metadata
  • Reject or refresh candidates whose versions do not match the published source
  • Group chunk-level candidates by source document when the interface needs document-level results