Define one retrieval unit for both indexes

Before creating embeddings or indexing text for BM25, define what a result represents. For many knowledge bases, that unit is a chunk: a bounded passage with enough surrounding context to answer a question, plus metadata that identifies its source document.

The dense and sparse representations should be derived from that same chunk record. If a semantic index contains one chunking strategy while the BM25 index contains another, a result from one path may not correspond cleanly to a result from the other. Ranking fusion then becomes an approximation over mismatched units rather than a comparison of alternate evidence for the same passage.

  • Choose a chunk boundary policy, such as headings plus paragraphs or a fixed token window with overlap.
  • Assign each chunk a durable source document ID and an ordinal or source-location field.
  • Store the text used for retrieval alongside metadata needed to display or fetch the source.
  • Use the same canonical chunk record as input to regional S3 Vectors and Quickwit BM25.

Make the ID stable, but make the content versioned

A useful chunk ID distinguishes identity from revision. Identity answers which logical passage this is; revision answers which indexed representation is current. Treating every edit as an entirely unrelated chunk makes it harder to replace stale entries and to understand why a result changed after a reindex.

One practical pattern is to build an ID from a stable document key and a deterministic chunk location, then carry a content version separately. For example, a documentation page might produce an identifier such as docs/getting-started#chunk-03, while a content hash or ingestion revision records the specific text currently indexed.

  • Keep the same chunk ID when a passage is edited in place and its logical location remains the same.
  • Update the revision, content hash, and indexing timestamp when text changes.
  • Create a new ID when a structural change means the passage is no longer the same retrieval unit.
  • Apply deletions and replacements to both dense and sparse indexes as one ingestion workflow.

Fuse rankings only after joining on the shared ID

Dense search can surface conceptually related passages, while BM25 can emphasize exact terms, names, codes, and uncommon vocabulary. When each result carries the same chunk ID, an application can merge the two ranked lists and make one candidate set for downstream use.

Reciprocal rank fusion is a simple starting point because it combines rank positions rather than assuming dense and sparse scores share a numeric scale. For a chunk d, a common form is RRF(d) = sum over result lists of 1 / (k + rank(d)). The constant k reduces the influence of very high rank positions; it should be selected and reviewed against representative queries rather than assumed to be universal.

  • Request a candidate list from each retrieval path and retain rank, chunk ID, and relevant metadata.
  • Deduplicate by chunk ID before presenting results or passing context to a later stage.
  • Log which path retrieved each final candidate to investigate unexpected matches.
  • Test exact-phrase, synonym, identifier, and multi-topic queries when reviewing the retrieval contract.