The production result

The primary run executed 1,000 hybrid search requests at 25 QPS through the production API. It completed with 1,000 successful HTTP 200 responses, a 0% error rate, 169.1 ms p50, 219.0 ms p90, and 333.0 ms p99. This is API-client latency measured from a Render one-off job in the same production environment, not a unit-test timer.

We also measured dense retrieval from outside Render against the public API. That run reached 392.4 ms p90 end to end, while the API's own reported dense execution p90 was 148.5 ms. Sparse retrieval measured 109.4 ms p90 and the earlier hybrid stage measured 193.1 ms p90 at 10 QPS. The 1,000-query run is the stronger hybrid result because it adds sustained traffic and a larger sample.

  • Hybrid: 1,000 requests, 25 QPS, p50 169.1 ms, p90 219.0 ms, p99 333.0 ms
  • Sparse: p90 109.4 ms at 10 QPS
  • Dense public API: p90 392.4 ms end to end
  • No retrieval errors in the measured sparse or hybrid stages

What was on the request path

A hybrid query has two independent retrieval paths. Dense retrieval goes to the index's regional Amazon S3 Vectors index. Sparse retrieval goes to the regional private Quickwit service, where BM25 and literal matching run against immutable S3-backed splits. The API validates access, filters and active versions, merges the result sets, and returns the selected evidence.

Those services do not depend on each other for one query. Treating them as serial work means hybrid latency becomes dense time plus sparse time plus merge time. The relevant optimization was not a new ranking algorithm. It was removing a wait that did not express a real dependency.

Three changes that reduced the tail

First, hybrid requests now start dense and sparse retrieval concurrently. The API waits for both responses only at reciprocal-rank fusion, where the dependency actually exists. This removes an entire remote wait from the normal hybrid path.

Second, sparse retrieval starts with a small candidate page for small top-k requests. Earlier behavior could request 250 candidates for a request that needed three results. The API still pages when tombstones, filters, or stale sparse versions require more candidates, so this is an overfetch reduction rather than a recall shortcut.

Third, lean SDK responses that explicitly set include_metadata to false no longer trigger a second Quickwit request only to decorate results with snippets. Metadata-rich console and assistant paths retain snippets; the low-latency API path avoids work the caller declined.

  • Parallel dense and sparse remote calls for hybrid retrieval
  • Candidate overfetch sized to the requested top-k before fallback paging
  • No snippet fetch for intentionally metadata-free responses
  • Version-ledger validation remains in the sparse path

We measured relevance before load

A latency improvement is not valid if it silently drops relevant candidates. Before the load stage, the runner creates disposable enterprise organizations and labeled vectors, then checks dense, sparse, hybrid, exact, and regex retrieval through the public API. Every mode returned its expected result at rank one: precision@1, recall@1, and F1@1 were all 1.0.

The test data is not customer data. The runner provisions temporary organizations, scoped keys, regional indexes, and sparse indexes, then removes those fixtures in a finally block. After the 1,000-query run, the control plane had zero residual load-test organizations.

What this target does and does not cover

The less-than-500-ms target is realistic for direct dense, sparse, hybrid, exact, and regex retrieval. It is not a promise for agentic search or Assistant RAG. Those modes may call remote embedding or generation models, so their total latency includes an external model round trip and should have separate SLOs.

Results will also vary with client geography, region selection, index size, filters, result shape, and concurrency. We publish the request count, QPS, percentiles, error rate, and relevance gate so the result is interpretable rather than a single best-case number. The operational rule remains simple: measure API p50/p90/p99 continuously, and keep quality checks beside latency checks.