perf(webapp,run-engine,database): resolve the newest worker and deployment by createdAt - #4452
Conversation
…yment by createdAt Ordering these lookups by id makes Postgres scan the primary key backwards betting on an early match for the target environment. When that bet loses the scan crosses a large part of the index. Ordering by createdAt then id keeps the lookup on a composite index instead. Adds (environmentId, createdAt) on WorkerDeployment, which had no matching index. BackgroundWorker already had one.
|
WalkthroughWorker and deployment resolution now orders records by descending 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…t fallback findCurrentWorkerDeployment resolved the caller-supplied client for the promotion read but used the module-level client for the fallback read, so callers passing a replica silently read the primary for that one query.
2 participants
Summary
Resolving "the most recent background worker for this environment" was ordered by
id, which Postgres serves by scanning the primary key backwards and betting it finds a row for the target environment early. For an environment with many worker versions whose rows are all old, that bet loses and the scan crosses a large part of the index, so a lookup that normally takes about a millisecond takes hundreds. The same shape appeared on the deployments table.These lookups now order by
createdAtthenid. On background workers that lands on the existing(runtimeEnvironmentId, createdAt)index with no schema change. Deployments had no matching index, so this adds one:The
idtiebreak costs an incremental sort over a singlecreatedAtgroup rather than a sort of the whole match set, so the ordering stays index-served.Also drops three
orderByclauses that were dead: twofindFirstcalls keyed by primary key, and one hydration read whose caller re-sorts the rows itself.Behaviour change
This changes which row counts as newest wherever
idorder andcreatedAtorder disagree: those lookups now answer with the genuinely newest row. Verified end to end against a live environment by forcing that disagreement and confirming a real run resolves to, and successfully executes, thecreatedAt-newest worker. Tests covering both resolvers were red before the change and green after.Rollout
The index build is the long pole and wants to go in on its own, so the migration and the code deploy should not be assumed simultaneous. Rollback is a revert plus
DROP INDEX CONCURRENTLY "WorkerDeployment_environmentId_createdAt_idx". No data migration and no coexistence concern, since nothing persists the ordering and old and new code can run side by side.Known limitation
The new index is keyed
(environmentId, createdAt), so the deployment fallback'stype = 'MANAGED'predicate is applied as a post-index filter rather than an index condition. For an environment whose newest MANAGED deployment sits behind many non-MANAGED ones, that scan still walks the intervening index entries. Left as-is on purpose: the high-volume query this index targets has notypepredicate, and the fallback only runs when the promoted deployment is not MANAGED. If that fallback ever becomes hot, a partial index on MANAGED is the better shape than widening the composite.Second change in here
findCurrentWorkerDeploymentresolved the caller-supplied prisma client and used it for the promotion read, but issued the latest-of-type fallback read on the module-level client. Callers passing a replica were silently reading the primary for that one query. The fallback now uses the caller's client too.This is a behaviour change, not just a tidy-up: the fallback read can now be replica-lagged for callers that pass a replica. That is the intended trade, because the previous mix could return a promotion read from a replica alongside a fallback deployment from the primary that did not correspond to it, and the promotion read (which serves the normal case) was already on the caller's client.