Treat the chunk ID as a retrieval contract

Assign an immutable canonical ID when a document is split into chunks. Store that same ID with the dense vector written to regional S3 Vectors and with the sparse document indexed in Quickwit BM25. The ID should identify a particular chunk revision, not merely its source document.

For example, an ID can combine a stable source identifier, a content revision, and a chunk ordinal: `handbook-42:r17:c003`. The exact format is less important than its determinism. Given the same source revision and chunking rules, your ingestion process should produce the same identity every time.

  • Use IDs that are unique at the chunk level, not just the document level.
  • Include or associate a source revision so updated content does not silently overwrite older chunks.
  • Keep the ID independent of embedding model names or index-specific implementation details.
  • Store the source document ID separately for grouping and display.

Fuse rankings by ID, not by text

When dense and sparse searches return results, merge them through the canonical chunk ID. This prevents duplicate answer candidates when both systems retrieve the same chunk and gives the application one place to combine rank signals.

A straightforward starting point is reciprocal rank fusion (RRF). For each result list, add `1 / (k + rank)` to the score for every chunk ID, then sort IDs by their accumulated score. RRF does not require dense similarity scores and BM25 scores to share a common scale, which is useful because those scores have different meanings.

  • Deduplicate before presenting context to a model or user.
  • Retain per-retriever rank and score as diagnostics, even if final ranking uses fusion.
  • Use a fixed `k` initially and evaluate changes with a representative query set.
  • Apply document-level diversity after fusion if many top chunks come from one source.

Make updates and deletes ID-driven

Stable IDs also simplify the less visible part of retrieval quality: keeping indexes aligned. When a source changes, compute the new chunk set, write the new revision, and remove chunk IDs that are no longer present. The same lifecycle operation should be reflected in both the dense and sparse retrieval paths.

Keep an ingestion manifest outside the indexes that records the source revision, chunk IDs, content hashes, and indexing status. If an indexing job is retried, the manifest lets the application determine whether a chunk is new, unchanged, replaced, or awaiting cleanup rather than relying on best-effort assumptions.

  • Use content hashes to avoid re-embedding unchanged chunks.
  • Make writes idempotent so retries do not create ambiguous records.
  • Track delete failures separately from write failures.
  • Periodically compare manifest IDs with IDs represented in each retrieval path.