Treat a chunk as the canonical retrieval record

A document is often too large to be the useful unit of retrieval. Split source material into chunks, then define each chunk as a durable record with an ID, text, source reference, position, and metadata. That ID should be assigned before indexing rather than generated independently by each retrieval system.

The same chunk ID should accompany the embedding sent to the dense index and the text sent to the sparse index. This does not require dense and sparse systems to store identical fields, but it does require their result sets to refer back to the same canonical chunk record.

  • Use an immutable chunk ID such as a source identifier plus a content-version and chunk position.
  • Keep source document ID, chunk ordinal, title, tenant or access scope, and updated-at data in the canonical record.
  • Store the chunk text or a resolvable text reference so returned IDs can be rendered consistently.
  • Version chunking logic when changing split rules to avoid ambiguous old and new chunk IDs.

Index dense and sparse representations from one ingestion event

Build an ingestion pipeline that produces the chunk record once, then sends the appropriate representation to each search path. The dense path receives an embedding associated with the chunk ID; the sparse path receives searchable text associated with that same ID. Talqora’s architecture combines regional S3 Vectors for dense search with Quickwit BM25 for sparse search, so this shared identity prevents the two paths from drifting apart conceptually.

Make updates idempotent. If a source changes, create or update the corresponding versioned chunks, index both representations, and retire records that are no longer valid. A retrieval system becomes difficult to trust when one index still returns content that the other index considers deleted.

  • Generate text, metadata, and embeddings from the same normalized chunk payload.
  • Apply the same tenant, authorization, and lifecycle filters to both retrieval paths.
  • Record ingestion status separately for dense and sparse indexing so partial failures are visible.
  • Use a reconciliation job to identify chunk IDs present in only one index.

Fuse ranks by ID, then inspect disagreement

Once both searches return the same chunk identifiers, hybrid fusion can happen in the application layer. Reciprocal rank fusion (RRF) is a useful starting point: each chunk receives credit based on its rank in each result list, and the credits are added. The method avoids assuming that a vector similarity score and a BM25 score are directly comparable.

The operational benefit is as important as the ranking benefit. When a result appears only in sparse retrieval, only in dense retrieval, or in both, engineers can inspect a single canonical chunk. That makes it easier to find chunking problems, missing metadata filters, vocabulary gaps, or embedding coverage issues.

  • Retrieve a candidate list from dense search and another from BM25 before fusion.
  • Deduplicate candidates by canonical chunk ID, not by displayed text.
  • Apply final access-control and freshness checks after fusion and before returning results.
  • Log each candidate’s dense rank, sparse rank, fused rank, and chunk version for debugging.