Start with one canonical document identity

Dense and sparse retrieval should refer to the same logical content using the same stable identifier. If a product page is represented by a vector record in one system and a differently named text record in another, joining results becomes an application-side guessing exercise. Use an immutable document ID that survives re-indexing, embedding-model changes, and text updates.

For Talqora deployments, this identity can anchor records stored for dense search in regional S3 Vectors and records indexed for sparse search with Quickwit BM25. The two retrieval methods may use different representations of the content, but the candidate they return should resolve to the same document ID and version.

  • Use a canonical ID such as a UUID or durable source-system key.
  • Keep a document version or content hash alongside the ID.
  • Treat chunk IDs as separate from parent document IDs.
  • Avoid using mutable fields, such as titles or paths, as join keys.

Define the candidate payload before tuning ranking

A retrieval response is more useful when every candidate includes the fields needed by the next stage. At minimum, that usually means the canonical ID, parent ID when chunking is used, source type, relevant filter metadata, and the retrieval score or rank. This lets an application merge dense and BM25 candidates without fetching the full document merely to identify what it found.

The payload should distinguish retrieval facts from presentation data. A rank, score, index name, and matched chunk ID help diagnose retrieval behavior. A title or display snippet may help a user interface, but it should not be the only data available for a downstream reranker, deduplication step, or audit log.

  • Return canonical ID, chunk ID, and parent document ID where applicable.
  • Include score or ordinal rank together with the retrieval method.
  • Carry filterable metadata consistently across dense and sparse indexes.
  • Record an index or content version to detect stale candidates.

Fuse ranks conservatively and inspect disagreements

Dense similarity scores and BM25 scores are produced by different ranking functions, so their raw numeric values should not automatically be added together. A safer initial approach is rank-based fusion: retrieve a bounded candidate list from each path, combine candidates by canonical ID, and assign credit based on each candidate’s position in its respective list. This avoids assuming that score scales are directly comparable.

Disagreement between the two paths is valuable diagnostic information. A BM25-only result may indicate an important exact term, identifier, or rare phrase. A dense-only result may reveal a semantic match that does not share vocabulary with the query. Logging these cases helps teams improve chunking, metadata filters, query construction, and relevance judgments before changing fusion weights.

  • Retrieve a fixed candidate depth from dense and sparse paths.
  • Deduplicate by canonical or parent document ID before final presentation.
  • Use rank-based fusion before attempting score normalization.
  • Log dense-only, sparse-only, and overlapping candidates for review.