Report worker boot failures on /health instead of exiting silently

[NathanTarbert]

The worker reports why a boot failed instead of dying silently, and a worker doing long work is no longer reported as stalled. This restores the /health rework that d0c6f99 deferred, with the three incident-class defects that deferral named fixed rather than re-landed.

Closes the worker half of #187's split (CPK-7926).

What is deliberately NOT restored

apps/worker/start.sh, the schema-drift guard and the repair migration stay exactly as they are on main. Those are the verified parts of the original PR, and d0c6f99 replaced the drift guard's regex classifier with the plain migrate diff --exit-code form for good reason — the classifier passed real drift, since ALTER TYPE ... ADD VALUE and ALTER COLUMN match neither pattern. Restoring b8a68e3 wholesale would have brought it back. apps/worker/Dockerfile is untouched too; its HEALTHCHECK port fix was kept on main.

The liveness model, which is the substance of this PR

Defect 1 was a healthy worker answering 503 "stalled". poll() stamps lastPollTime and then awaits the jobs it claimed, so the stamp freezes for as long as the work takes. b8a68e3 compared it against STALE_POLL_MS = 60s, so a 300s HUBSPOT_SYNC read as wedged and the container probe killed the worker mid-job.

Bounding the poll instead does not work, and this is the part worth reading. My first attempt at this fix bounded an in-flight poll at max(jobTimeouts) + grace = 360s. That is also wrong, because claimJobsByType awaits each type's batch sequentially:

for (const { type, available } of typesWithCapacity) {
    const jobs = await this.claimJobsForType(type, limit);
    if (jobs.length > 0) {
        await Promise.allSettled(jobs.map((job) => this.processJob(job)));

One poll is therefore the sum across types, not the max — 930s against this worker's ten registered types, every job inside its own timeout. Same incident as the one being fixed, reached by different arithmetic.

So poll duration is not the signal at all:

• overdueJobCount — in-flight jobs past their own timeout plus a grace. Needs no scheduling arithmetic, so it cannot drift out of step with how poll() batches.
• CLAIM_STALL_MS — bounds a poll with nothing claimed, which is the one case no job can be blamed for.
• lastPollCompletedAt — measures the gap between polls, which is what STALE_POLL_MS was always trying to measure.

maxJobTimeoutMs was in an earlier revision of this branch and is gone; it existed only to support the bound that was wrong.

Defect 3, and a variant of it from the other end

Number(process.env.X ?? default) collapsed to 0 on a cleared variable: ?? guards undefined and null, and Number('') is 0, not NaN. A cleared SHUTDOWN_WATCHDOG_MS became a 0ms watchdog forcing exit(1) on every SIGTERM mid-drain. resolveDurationMs replaces it.

It also bounds the top, which the original did not: setTimeout clamps anything above 2³¹−1 to 1ms, so an operator reaching for "effectively never" got "immediately" — the same defect from the other side. (Same shape as the SHADOW_MODE fail-open in #233, and worth noticing that this class keeps recurring.)

resolvePort is now strict and fatal on an explicitly-set but unusable value. Number.parseInt reads a numeric prefix, so "3003abc" silently bound 3003 and "0" bound an OS-assigned ephemeral port. The Dockerfile probes ${PORT:-${HEALTH_PORT:-3003}}, and shell :- substitutes only for unset or empty — so serving on a fallback guarantees a probe that can never connect, a container dead in ~90s, and /health answering 200 the whole way. A wrong port cannot be reported over that port, exactly like EADDRINUSE, so it fails closed.

Honesty fixes

failFatally now stops the scheduler and worker before exiting. It was leaving every job claimed in its 5s window at status='PROCESSING', and nothing requeues a stale lockedAt, so those jobs were lost rather than retried — the hazard startWorker's boot-after-shutdown guard exists to prevent, reached by another path. A stale-lockedAt requeue sweep is worth its own change; abrupt exits are not avoidable in general.

A post-boot crash now reports as crashed, not failed. The failed-boot log advice points at schema.prisma and the drift guard, so filing a handler's stray rejection under it sends whoever is paged to audit migrations for a bug in an AI handler.

Config warnings ride on /health instead of dying in a boot log. The watchdog one is the sharpest: it prints at boot and only bites at the next deploy, potentially weeks later, when nobody is looking at that line.

Verification

turbo build / typecheck / lint / test 10/10, lint 0 errors. apps/worker 22 → 70 tests. packages/outpost 1196 → 1208. Prettier clean on every changed file. Also ran the root vitest.config.ts — 4 files / 64 passing — because no turbo task runs it, so scripts/__tests__/** is otherwise green-by-absence in CI.

Mutation-tested, since vacuous tests are what sank the first attempt:

Mutation	Tests failed
old max(jobTimeouts) bound model restored	7
overdue check moved after the timing branches	1
resolvePort accepts 0 again	1
resolvePort back to lenient parseInt	3
duration upper bound removed	3
crash classified as failed	2
failed boot gets a second exit timer	1
overdue grace removed	1

Two guards I removed, and one I kept without coverage

Mutation testing showed no input could reach either of these, so they are gone rather than sitting there looking like protection: a poll re-entrancy check (poll() only ever runs from a timer armed at the end of a poll, so it cannot overlap itself) and a poll-generation token (its only scenario needs a stop()/start() overlap nothing performs, and it mutually masked with the !running check, so no single mutation could fail).

reschedule() keeps its !running guard. Being straight about it: removing that guard leaves the suite green. poll() already returns on !running, so the guard prevents a queued timer handle rather than lost work, and after processing nextPollDelay is 0 — so the re-armed timer fires on the next flush either way. The test comment says so rather than implying coverage. It is kept because a handle queued behind a resolved stop() is worth preventing at the source.

What is not covered

• index.ts has no direct tests. It is a top-level-await module with import-time side effects, so its boot behaviour cannot be exercised in-process. classifyFatalError and the boot-state logic are extracted into health.ts for that reason and are tested there, but the wiring itself is not. Extracting startWorker to accept injected Worker/Scheduler factories would fix that and is worth doing separately.
• No startup assertion and no dashboard surface. Nothing announces the resolved state at boot beyond the log, and /health is still the only place to read it.
• railway.toml is untouched. An earlier revision of this branch lowered healthcheckTimeout to 180; I reverted it. The Dockerfile budgets --start-period=120s for two Prisma CLI invocations before anything binds, which leaves only 60s of headroom, and the deploy this unblocks is staging — the environment with the largest pending-migration backlog. Timing a real migrate deploy against that backlog should come before narrowing the window.
• runWithTimeout is a Promise.race that does not cancel the handler, so an orphaned handler keeps running and holding its connection after its job is marked failed. activeJobCount therefore understates real work. Pre-existing, and overdueJobCount is measured against the tracked entry rather than the orphan, so this PR does not depend on it.
• /health is unauthenticated and spreads the whole worker snapshot, including registeredHandlers. main's handler did the same, so this is pre-existing — flagging it because this is the change that establishes what the endpoint is allowed to say.

Review note

The earlier revision of this branch went through a review round, which is what caught the 360s bound described above. The rewrite that replaced it has not been reviewed, so a pass over overdueJobCount and CLAIM_STALL_MS specifically would be worth having before merge.