fix(queue): fence job claims and drain in-flight work on shutdown - #224
fix(queue): fence job claims and drain in-flight work on shutdown#224jerelvelarde wants to merge 7 commits into
Conversation
Third split of #187, worker correctness only. `/health` and railway.toml are deliberately NOT here — see below. A claim is now fenced by `Job.claimToken`. `updateJobProgress` takes the token and writes through `updateMany` gated on `status: 'PROCESSING'` plus a matching token, so a stale execution — one whose job was reclaimed after its lock went stale, and which is still running because a timeout race does not cancel the handler — can no longer clobber the live claim's progress. The old `prisma.job.update({ where: { id } })` had no way to tell the two apart. Also here: reclaiming stale PROCESSING jobs against each type's own timeout rather than one global value, a recovery grace before crash-abandoned claims count toward dead-letter, and a shutdown that shares one drain across repeated stop calls and waits for an in-flight poll instead of racing it. `queue.test.ts` is included, which the split derivation did not predict. It was excluded from the third PR's file list because it also appears in #191 — but #191's edit is only the job-type counts, while the worker tests this branch needs live in #187's version of the same file. Without them the suite fails 11 tests on `prisma.$executeRaw is not a function`: the new `reclaimStaleJobs` issues raw SQL that the existing prisma mock never had to provide. WHY /health IS STILL DEFERRED, a third time: Four reviewers previously found the /health rework makes a healthy worker report 503 and get restarted mid-job, and the stated condition for landing it was "the poll loop is fixed, since the poll loop is what makes it wrong". That condition is NOT met by this branch. `claimJobsByType` still awaits `Promise.allSettled` over each type's batch, so `poll()` blocks for the full duration of the work it dispatched and `lastPollTime` does not advance. `buildHealthResponse` compares it against `STALE_POLL_MS` (60s) with no exemption for active jobs, and AI_RESPONSE has a 120s timeout — so a worker doing exactly what it should would answer 503 "stalled" after 60s and Railway would restart it mid-job. Closing that needs one of two changes, neither of which belongs in this PR: dispatch without awaiting the batch, or exempt active work from the stale-poll check. #182 and #184 stay open. `health.ts` also wires through apps/worker/src/index.ts, which #191 already modifies. Advances #153, #181 Verified: packages/outpost — 61 files, 1017 tests pass.
CPK-7925 3. Open + merge the third split PR — worker correctness (unblocks response quality)
The third of the three PRs split out of #187. Not yet opened — #191's description names it as "third (worker correctness) to follow." Scope — derived from the files #187 carries that neither split doesRoadmap issues it should close or advance
Worth confirming that list against the actual diff when the PR is opened rather than assuming it — the file overlap is strong evidence of scope, not a promise of what the code does. Why this is the response-quality gate
Until this lands, work on any of the three either conflicts with this branch or gets rewritten by it. One thing to decide when opening it: whether the CPK-7982 Review PR #224 — fence job claims and drain in-flight work on shutdown
CopilotKit/outpost#224 by jerelvelarde, opened 2026-08-19 12:48Z off This is the worker half of the CPK-7925 split (#223 is the generator half). It should close or advance #181 (
|
The migration added the column; schema.prisma never declared it, so the migrations no longer reproduce schema.prisma from scratch and CI's drift gate correctly failed with `[-] Removed column claimToken` (exit 2). Exactly the class of defect that gate exists to catch — a migration whose effect is real but invisible to the schema — so it working here is the system behaving as designed, not an obstacle. Caught only in CI because the check needs a live Postgres, which this environment has neither Docker nor a local server for. `prisma validate` and the full package suite both pass locally and neither can see this.
|
Hey @jerelvelarde, I deleted my two earlier comments on this PR and replaced them with this one. They were long, they buried the actual blockers, and a few of my claims turned out to be wrong. I re-ran the review with mutation testing enforced (delete the fix, confirm the test fails) and it changed several answers, including retracting some of mine. Everything below is either executed or explicitly marked as reasoning. Verdict: NEEDS CHANGES. Three blockers. Everything else here is either approved or filed as non-blocking, so you can ignore the rest until the three are in. Approved: don't change theseThe claim-fencing mechanism is correct, and it's the hard part of this PR. The The migration is safe and additive. Nullable with no backfill is the right call (see the retraction below). It must deploy ahead of the code, and it's a no-op for the running fleet, so that ordering is free. BlockersB1. Reclaim resets the retry backoff
Walk it: W1 claims a Fix: B2. The reclaim window comes from the wrong worker
Rolling deploy, two replicas: W_new has Fix: persist the granted deadline at claim time ( B3. A fence rejection is silent on every path
Two consequences. In B2's interleaving a job completed, the row says Fix: Mutation confirming this is untested: changing both Non-blocking, your call on timing
Tests: this is the part I'd fix firstFive of thirteen mutations survived, and every survivor lands on either the reclaim SQL's semantics or the code this PR edits.
The first two mean the reclaim's headline behavior, per-type timeouts, can be deleted with nothing failing. The third is the one I want to flag hardest, because I praised this line in my deleted comment: the test named Root cause: Suggested additions: assert Retracting five things I said earlierWorth being explicit, since you may have started on some of these:
And I had two backwards on severity: the backoff reset and the config-skew window were filed as "smaller" and are the two blockers. Evidencevitest 4.1.4, single resolution from the lockfile, Base Not verified, stated plainly: no Postgres or Docker in the review environment, so the reclaim SQL was never executed and B1 and B3 are both small. Happy to take them as patches if you'd rather not context-switch, and B2 is the one worth talking through first since |
Three review blockers. B1 — reclaim reset the retry backoff. The requeue branch set `runAt = NOW()` while `handleFailure` uses `calculateBackoff`, so two paths that both mean "this attempt did not finish" disagreed. A worker SIGKILLed on Railway grace expiry got its job reclaimed, re-claimed under a second later, killed again — five attempts and DEAD_LETTER inside ~25 minutes with zero spacing, and a crash-looping worker driving every job there at full speed. Reclaim now mirrors `calculateBackoff` in SQL (base * 2^attempt plus jitter under a ceiling, on `attempts + 1` to match `handleFailure`). B2 — the reclaim window came from the wrong worker. `reclaim_after_ms` was derived from the *observing* process's `jobTimeouts`, and nothing on the row recorded the timeout the claim was granted under. Mid-rolling-deploy, a replica without an entry for a long-running type falls back to `defaultTimeoutMs` and reclaims a claim that is still live; the original handler then finishes, its fenced write matches zero rows, and completed work is silently discarded while the job runs a second time. Adds `Job.lockUntil`, written at claim time from the claiming worker's own config, looked up per-type from a jsonb map with the worker default as fallback. Reclaim compares against that. The row is now self-describing, which also means the type predicate is gone: a crashed claim of a type this replica does not handle used to stay PROCESSING forever, never reclaimed and never dead-lettered. Rows claimed before this deploys have a NULL `lockUntil` and fall to an absolute 15-minute ceiling — well above the largest configured timeout — rather than a guessed deadline, since the guess is the bug being removed. B3 — a fence rejection was silent on all four paths. `count === 0` is the event this PR exists to produce and it emitted nothing; the two `count > 0` checks read the count only to *suppress* a log, so a fenced retry or dead-letter also lost the failure that caused the timeout. All four now warn, naming the job and the token, and the retry/dead-letter paths carry the original error into the fence warning. Also, unprompted by a blocker but introduced by this PR: the reclaim sweep sits ahead of every claim in the same try with `lastPollTime` already set, so a failing sweep stopped all claiming while health still reported green — a total outage with nothing to restart it. Now isolated in its own try/catch. Migration: renamed from `20260811220000` to `20260820120000`. The old stamp sorted before six already-applied migrations, so applied order no longer matched lexicographic order. Free to fix now because this has never been applied outside CI. Adds `Job_status_lockUntil_idx` for the sweep, which runs once per poll interval per replica and was served by no existing index. Plain, not CONCURRENTLY: Prisma applies each migration in a transaction, which forbids it. Tests: every blocker now dies under its mutation, where none did before. Restoring `runAt = NOW()` fails 1, reverting reclaim to the observer's config fails 3, dropping `lockUntil` from the claim sites fails 1, and `count >= 0` — the exact mutation from the review that used to pass the whole suite — fails 4. 1087 tests in the package, prettier clean, typecheck unchanged from base (same 49 pre-existing). Not verified locally: no live Postgres or Docker in this environment, so the reclaim and claim SQL is unexecuted. CI's `migrate deploy` will exercise the migration and the drift gate, but the queries themselves are mocked in tests and first run for real in staging.
|
All three blockers fixed in B1 — reclaim reset the backoffConfirmed and fixed. Your walkthrough is right and I hadn't thought about it at all: two paths that both mean "this attempt did not finish" have to space retries the same way, and I'd written one of them as Reclaim now mirrors B2 — the window came from the wrong workerThis is the finding I'd have been least likely to find myself, and it's the worst of the three: silently discarded completed work with the job running twice. Took your fix. One thing I went further on than you asked, and I'd like you to check it: once the row carries its own deadline, the type predicate has no reason to exist, so I dropped it. That folds in your first non-blocking item — a crashed claim of a type this replica doesn't register used to sit Legacy rows: B3 — silent fencesConfirmed, including your mutation: Also taken: the reclaim-gates-claiming outageYou filed this non-blocking and I'd rather not ship it — it's introduced by this PR (there was no reclaim before), and "total outage reporting healthy, one log line per second, no restart" is the worst failure mode in the list. Own try/catch, so a failing sweep can't stop claiming. Recovering abandoned work is a nice-to-have; claiming new work is the job. Also taken: the migration rename and the indexRenamed Index added as Not taking here, with reasonsThe concurrency accounting being unreachable, and narrowing the #153 claim. Both correct. Unregistered-type destruction at Timeout doesn't cancel, and the drain has no deadline. Agreed, and thank you for stating the honest contribution plainly — the real shutdown path is still crash-abandonment plus reclaim, and the fence makes that safe at the row level rather than preventing it.
Side effects outside the row. This is the honest limit of the PR and I don't want it lost, so it's going in the body rather than a follow-up: the fence is a DB guard, and in a B2-style interleaving the first execution's external effects have already happened. What I could not verifyNo Docker and no local Postgres in this environment, so the reclaim and claim SQL is unexecuted. The tests mock Re-review when you have a moment. |
|
Hey @jerelvelarde, Verified
Head 1087, base 1081, delta +6. Typecheck 49 at head and 49 at base, identical, all pre-existing. Prettier clean. The backoff mirror is exact. This is the item I most expected to find a divergence in and there isn't one. Term by term against And Two things before merge1.
|
NathanTarbert
left a comment
NathanTarbert
left a comment
There was a problem hiding this comment.
All three blockers verified fixed, and every claimed mutation dies at the claimed count. Two items before merge, both small: the new index can't serve the sweep's predicate (the CASE isn't sargable), and the reclaim try/catch you added unprompted is the one new behavior with zero coverage. Details in the comment above.
Answers the two merge blockers on #224 plus the four adjacent items. 1. `Job_status_lockUntil_idx` could not serve the query it was added for. The deadline test was a `CASE` over `lockUntil`, which is not sargable, so the planner took the `status` prefix and filtered every PROCESSING row. Now a disjunction, with the interval arithmetic on the right-hand side — both are required, since `lockUntil + interval < NOW()` is still non-sargable in disjunctive form. Recorded next to the index in schema.prisma so a future edit does not silently undo it. 2. The reclaim try/catch was the only new behaviour with no coverage. Deleting it passed the whole suite. Now pinned by asserting the claim still runs after a rejected sweep, and that the failure is logged. Also taken from the review: - All five release paths clear `lockUntil`. Inert while the predicate gates on `status = 'PROCESSING'`, but the first future path that sets PROCESSING without writing a fresh deadline would inherit one already in the past. - `worker.ts:461`, the no-handler tombstone, was the fifth fenced write and the last silent one. Now reports through `warnIfFenced`. - `"attempts" = job."attempts" + 1` is asserted as a `SET` fragment. The old `toContain('job."attempts" + 1')` was satisfied by the string's other occurrences, so a job that never accrues an attempt — and so never reaches DEAD_LETTER — went unnoticed. - The type-blindness guard asserts no `job.type` anywhere in the sweep. `AND job.type = ANY(...)` walked past the old single-spelling check. - The legacy ceiling is checked against this worker's own timeouts at construction. It stays a fixed constant — deriving it from the observing worker's config is the bug `lockUntil` removed — but fixed means it does not follow `jobTimeouts` upward, and the two live in different packages. The docstring also now names ACCOUNT_SCORING, which ties HUBSPOT_SYNC at 300s. Call sites enumerated: - `warnIfLegacyCeilingTooLow` (added, private): one caller, the constructor. - `warnIfFenced` (existing): one new call site; the other four unchanged. - `LEGACY_RECLAIM_CEILING_MS` / `STALE_RECOVERY_GRACE_MS`: new references in the guard and the predicate; existing uses unchanged. - `Job.lockUntil`: every writer is a claim (2), the reclaim (1), or a release (5). No other writer exists in the repo. Verification: 1113 tests pass (62 files). Eleven mutations run, each killed by exactly the test that names it — including deleting the try/catch, reverting the predicate to a CASE, dropping the attempt increment, re-adding a type predicate, and removing `lockUntil: null` from each release path. Queue typecheck 49 errors at head and 49 at base, identical and all pre-existing (unbuilt workspace). Prettier clean, no collateral reformatting.
… DB errors
A seven-agent review of the previous commit found one surviving mutation that
matters more than everything else here, plus a set of silent paths around it.
## The surviving mutation
Deleting the `lockUntil` SET from `claimJobsForType` — and from that one query
only — left all 1113 tests green. Production sets `concurrencyByType`
(apps/worker/src/index.ts), so `claimJobsForType` is the only claim path it ever
takes. Every job it claimed would have carried `lockUntil = NULL` and fallen into
the 15-minute legacy branch forever, which is the exact failure this PR exists to
remove. The only covered claim path was the one production never runs.
Two claim queries now carry byte-identical SET clauses duplicated by hand, so
there is a test asserting they are equal as well as one covering the per-type
query directly. Both die under the mutation above.
## Bookkeeping is not the work
- The COMPLETED write sat inside the handler's own `try`, so a Prisma blip on it
routed into `handleFailure`: the row went back to PENDING carrying a database
error as the *job's* error, and the retry re-ran every side effect the handler
had already produced. That is the duplicate execution the claim token exists to
detect, manufactured from a bookkeeping failure. The write now sits outside, and
a failure there leaves the row for the sweep and says so.
- `Promise.allSettled` discarded its results and `processJob` has no outer catch,
so a rejection there produced no output at all — not the Prisma error, and not
the handler failure it was recording. Rejections are now reported.
- `updateJobProgress` could reject into a handler that awaits it, failing a job
that was running fine. Progress is telemetry; it can no longer fail the job.
## The sweep had no voice and one blind spot
- Its row count was computed once per poll on every replica and thrown away. It
is the earliest signal that replicas are dying, so it is logged when non-zero.
- It ran behind the capacity check, so a saturated replica stopped recovering
abandoned work exactly when the backlog was largest. Moved ahead of it.
- A PROCESSING row with neither `lockUntil` nor `lockedAt` matched no arm and sat
there forever — unreachable from the two claim paths, which is precisely the
assumption a last-resort sweep should not make. Third arm on `updatedAt`.
- It overwrote `error` unconditionally, including on the arm that dead-letters,
destroying the failure that actually killed the job at the point an operator
goes looking. Now appended, bounded with `left(…, 200)`.
- `POWER(2, attempts + 1)` is unbounded and `maxAttempts` is settable per row.
float8 overflow would abort the sweep for every row, not just the offending
one — and JS saturates gracefully there, so the mirror diverged at the extreme.
Exponent clamped at 30, which saturates to BACKOFF_MAX_MS exactly as JS does.
## Claims that were not true
- The fence message asserted "this job ran more than once". `count === 0` proves
only that the claim was lost; the sweep may have dead-lettered the row, in which
case it will never run again. Reworded to what is known.
- A missing `claimToken` would silently unfence every write in `processJob`,
because Prisma drops a `where` key whose value is `undefined`. Such a row is now
refused and left for the sweep rather than run unfenced.
- The sargability comment claimed both arms kept their column bare "for the index".
Only the first arm can use it; `lockedAt` is not in the index. Corrected, and the
test assertion that pretended to guard it is gone.
- `buildHealthResponse` does not exist. The method is `healthCheck()` — and the
health server answers 200 unconditionally without consulting it, so the outage
the catch guards against would not have been restarted either. Said plainly.
- The migration claimed it "must deploy ahead of the code". `start.sh` runs
`migrate deploy` at container start, so it ships with the code; what makes it
safe is the additive-nullable shape, not sequencing.
- The index migration is split out. Prisma wraps each file in one transaction, so
keeping it with the ADD COLUMN held that statement's ACCESS EXCLUSIVE across the
index build. Split, the ALTER commits first and the build takes only SHARE.
No `lock_timeout`: a timeout aborts the migration into the P3009 state
`start.sh` exists to diagnose, and waiting is the more recoverable failure.
## Tests were checking the mock against itself
`vi.mock('@copilotkit/outpost/shared')` re-declared MAX_JOB_ATTEMPTS,
BACKOFF_BASE_MS and BACKOFF_MAX_MS as literals, so every assertion written
against those numbers asserted nothing: editing the real constants left the file
green. They now come from `vi.importActual`. Also `$queryRaw` was never re-armed
between tests, so a test that set no claim result inherited the previous one's.
Call sites enumerated:
- `processClaimedJobs` (added, private): both claim paths, no other caller.
- `handleFailure` (added a `jobType` parameter): two callers, both in
`processJob`, both updated. Private, no out-of-class caller.
- `updateJobProgress` (behaviour changed, signature unchanged): every caller
reaches it through `context.reportProgress`; all of them await it, which is why
the change matters and why none needed editing.
- `Job.lockUntil`: unchanged from the previous commit — two claims, the sweep,
five releases.
Verification: 1125 tests pass (62 files). Fifteen mutations run, each killed by
exactly the test that names it — including deleting `lockUntil` from the
production claim path alone, letting the two claim clauses diverge, putting the
completion write back inside the handler's try, dropping the allSettled report,
moving the sweep back behind the capacity check, and editing the real constants
to catch mock drift. Queue typecheck 48 errors against a 49 baseline, all
pre-existing. Prettier clean, no collateral reformatting.
|
Both merge items fixed in The two you asked for1. 2. The outage guard had no coverage. Confirmed — deleting the Also taken: Then the review round found a surviving mutation that matters moreDeleting the
The two claim queries carry byte-identical SET clauses duplicated by hand, so there is now a test covering the per-type query directly and a second asserting the two clauses are equal. Both die under that mutation. (I looked at hoisting the fragment into one place; Silent paths around it, all fixed
Claims of mine that were not true
Tests were checking the mock against itself
On the index itself — your call, not mineTwo reviewers independently argued the index may not earn its keep: I have also split the index into its own migration — Prisma's per-file transaction meant the build was happening under the Numbers: 1125 tests, 62 files. Fifteen mutations in the second round, all killed. Queue typecheck 48 against a 49 baseline — one lower, because Still unexecuted, and I want to be as plain about it as you were: no Postgres here, so the reclaim and claim SQL has still never run. The third arm, Filed rather than fixedSix findings from the round are real, load-bearing, and belong to PRs we have already agreed are separate — I will open issues rather than smuggle them in here. The one I would look at first is deterministic type starvation in |
|
Went through this one against current head Every item from that review is closed, and I checked each by mutation rather than by reading:
Baseline at head is 62 files / 1125 tests, all passing. So this needs a re-review to clear the standing changes-requested, and a merge of Two notes for that pass. The body still says 62 files / 1087 tests and carries the 1087-era mutation table, where head is 1125 — worth refreshing since we've been holding other PRs to that. And the reclaim/claim SQL still hasn't executed anywhere: no Postgres in CI or locally, so the three-arm disjunction, |
2 participants
Third split of #187 — worker correctness.
generator.tswent out separately as #223;/healthandrailway.tomlare deliberately not here (see the last section, it's the important one).Advances #181, and half of #153: the fence guarantees one row's writes belong to one execution. It does nothing about two different
AI_RESPONSErows for the same ticket — different ids, different tokens, both proceed. That half is #191'sMessage.ticketId_responseKeyconstraint and theresponseStatemachine, not this.Claims are now fenced
updateJobProgresstakes aclaimTokenand writes throughupdateManygated onstatus: 'PROCESSING'and a matching token, backed by a new nullableJob.claimTokencolumn.The old
prisma.job.update({ where: { id } })had no way to distinguish the live claim from a stale execution — and stale executions genuinely exist here: a job whose lock went stale gets reclaimed, while the original handler is still running, because a timeout race in this codebase does not cancel the handler it raced. That older execution could keep writing progress over the new claim's. Now it writes zero rows.Also in scope:
stop()calls, and it waits for an in-flight poll rather than racing it.queue.test.tsis included, and the split derivation didn't predict thatWorth flagging, because it's a hole in how the third PR's scope was derived.
queue.test.tswas excluded from the file list on the grounds that it "also appears in #191" — but the two changes are unrelated:toHaveLength(6)→(7),(10)→(11)).Without them the suite fails 11 tests on
prisma.$executeRaw is not a function— the newreclaimStaleJobsissues raw SQL the existing prisma mock never had to provide. So "appears in another split" wasn't sufficient grounds to exclude it; the file needed the union, not either version.The five tests it adds are the ones that make the above claims falsifiable rather than asserted: stale-reclaim per type timeout, old execution not clobbering a reclaimed claim, crash-abandoned claims respecting the grace, drain shared across repeated stops, and no claiming after shutdown begins.
mainand therefore holds6/10for those two assertions. If #191 lands first, this needs those two lines bumped to7/11on rebase. Two lines, but it will conflict rather than merge cleanly, so it's worth knowing in advance.Why
/healthis deferred a third timeI went in expecting to include it and changed my mind on the evidence.
The stated condition for landing the
/healthrework was that it "cannot land until the worker poll loop is fixed, since the poll loop is what makes it wrong" — four reviewers having found it makes a healthy worker report 503 and get restarted mid-job. That condition is not met by this branch.claimJobsByTypestill doesawait Promise.allSettled(...)over each type's batch, sopoll()blocks for the full duration of the work it dispatched andlastPollTimestops advancing.buildHealthResponsecompares that againstSTALE_POLL_MS(60s) with no exemption for active jobs, andAI_RESPONSEcarries a 120s timeout.So a worker doing exactly what it is supposed to be doing would answer
503 "stalled"after 60 seconds, and Railway would restart it mid-job. The failure mode the reviewers described is still live.Closing it needs one of two changes, neither of which belongs in a PR about claim fencing:
#182 and #184 therefore stay open. Separately,
health.tswires throughapps/worker/src/index.ts, which #191 already modifies — so even a correct/healthwould want to follow #191.Verification
packages/outpost: 62 test files, 1087 tests passing.Every blocker from review is pinned by a mutation that used to pass:
runAt = NOW()lockUntilfrom the claim sitescount > 0→count >= 0Not verified: no Docker or local Postgres in this environment, so the reclaim
and claim SQL is unexecuted. Tests mock
prisma, so CI doesn't run it either —migrate deployexercises the migration and the drift gate, but the queriesfirst run for real in staging.
The limit of a database fence
The fence is a DB guard, so in an interleaving where a live claim gets reclaimed,
the first execution's external side effects have already happened.
lockUntilmakes that interleaving much rarer, but rarer is not impossible.
AI_RESPONSEis covered by #191's response-key constraint.ESCALATION,HUBSPOT_SYNCandTRACKER_SYNCare not, and this PR is what introducesautomatic re-execution — so they need their own idempotency keys. Filed rather
than fixed here.
Local note: a fresh worktree needs
pnpm installandnpx prisma generate --schema=db/prisma/schema.prisma, orscheduler.test.tsfails to load on.prisma/client/defaultand looks like a real failure.Migration hazards — read before pulling this branch
If you ran
migrate devwhile sitting on this branch, you have a stale migration applied. The folder was renamed20260811220000_add_job_claim_token→20260820120000_add_job_claim_fencing(the old stamp sorted before four already-applied migrations).migrate deploy/migrate devwill fail on any database that recorded the old name. Reset your local database, orprisma migrate resolve --rolled-back 20260811220000_add_job_claim_token. Ephemeral CI databases are unaffected.The index now lives in its own migration.
20260820120100_index_job_status_lockuntil. Prisma wraps each migration file in one transaction, so keepingCREATE INDEXalongside theALTER TABLE ADD COLUMNheld that statement'sACCESS EXCLUSIVElock across the index build and blocked reads for the duration. Split, theALTERcommits first and the index build takes onlySHARE: writes wait, reads do not. Deliberately nolock_timeout— a statement that times out aborts the migration into thefinished_at = NULLstateapps/worker/start.shexists to diagnose, and waiting is the more recoverable failure of the two.Worth checking the staging
Jobrow count before this promotes to prod.CONCURRENTLYis impossible inside Prisma's transaction, so the build is a plain one. Sub-second on a tableJOB_CLEANUPkeeps small; a queue stall ifJobis ever large.What only staging can settle
The reclaim and claim SQL is still unexecuted — the tests mock
prisma, so CI does not run it either.migrate deploywill exercise the migrations and the drift gate, but the queries themselves first run for real in staging. Specifically unverified: whether the disjunction's planner actually usesJob_status_lockUntil_idx, whether(jsonb ->> "Job".type)::double precisionand<param> * INTERVAL '1 millisecond'resolve under Prisma's parameter typing, andLEAST(<float8>, $n)against a Prisma-typed integer parameter.