Skip to content

[world-postgres] Add a producer role that enqueues without a runner or startup recovery - #3969

Open
Ehco1996 wants to merge 1 commit into
vercel:mainfrom
Ehco1996:feat/world-postgres-producer-role
Open

[world-postgres] Add a producer role that enqueues without a runner or startup recovery#3969
Ehco1996 wants to merge 1 commit into
vercel:mainfrom
Ehco1996:feat/world-postgres-producer-role

Conversation

@Ehco1996

@Ehco1996 Ehco1996 commented Sep 4, 2026

Copy link
Copy Markdown

Description

A process that only ever enqueues cannot avoid getting a consumer. queue() begins with await start(), and start() both starts a graphile-worker runner and calls reenqueueActiveRuns(). In the common "API process starts runs, worker process executes them" topology — where only the worker compiled the "use workflow" / "use step" code and serves .well-known/workflow/v1/* — both of those are unwanted in the API. Its runner claims due jobs and then fails their HTTP delivery whenever the worker has no ready endpoint (a rolling update, for instance), manufacturing failures the worker's own runner never produces, because that one dies together with its executor and leaves the job waiting. And every API replica runs full startup recovery, so N replicas rolling means N re-enqueues of every active run on top of the worker's own.

This adds role?: 'producer' | 'worker' to PostgresWorldConfig, defaulting to 'worker' — today's behaviour, no change for existing users. WORKFLOW_POSTGRES_ROLE is read lazily as a fallback, matching how the package's other options pick up environment configuration when it is selected through WORKFLOW_TARGET_WORLD. Under 'producer', start() still creates the worker utils, runs workerUtils.migrate(), and runs the pg-boss job migration, so a producer can enqueue into a fresh database; what it never does is call graphile's run() (none of the setupListeners / startRunnerWhenExecutorIsReady / deferRunnerStart paths) or reenqueueActiveRuns(). queue() is untouched, and close() already tolerates a null runner, so it ends only what was opened. Nothing about the recovery predicate changes: this removes N of the N+1 recovery sources behind #3119 and #3758 for this topology, and leaves the remaining one — the worker's own — exactly as it is.

How did you test your changes?

Unit tests in the two existing world-postgres suites, which already mock graphile-worker, so "no runner" is asserted directly on run() rather than inferred from a timeout.

src/reenqueue.test.ts (World level):

  • starts a graphile runner and recovers active runs in the default role — pins the unchanged default.
  • does not start a graphile runner in the producer rolerun() not called, migrate() still called.
  • does not re-enqueue active runs in the producer role — two active runs listed, no addJob.
  • still enqueues messages in the producer roleworld.queue(...) lands a workflow_flows job, still no runner.
  • reads the producer role from WORKFLOW_POSTGRES_ROLE.
  • ignores an unrecognized WORKFLOW_POSTGRES_ROLE — falls through to worker.
  • closes only what a producer openedrunner.stop() not called, workerUtils.release() called once.

src/queue.test.ts (queue level):

  • never starts a runner in the producer role, even when a local executor is reachable — a real loopback HTTP server is listening and PORT is set, so the unfixed code starts the runner synchronously inside start(); the producer starts none, enqueues fine, and the server receives no request.

All six new cases fail against unmodified main and pass with the change.

Commands run locally:

  • pnpm exec biome ci --max-diagnostics=200 — exit 0 (warning count unchanged; the only warning in the touched files is the pre-existing createTaskHandler complexity one).
  • node scripts/check-no-unrun-tests.mjs — pass.
  • node scripts/check-changesets.mjs — pass, 1 pending changeset, 1 package released.
  • tsc --noEmit in packages/world-postgres — clean. (pnpm turbo typecheck could not run end to end here: the dependency graph builds @workflow/swc-plugin, which needs a Rust toolchain with the wasm32-unknown-unknown target that this machine does not have.)
  • vitest run src/ in packages/world-postgres — 4 files, 35 tests passed.
  • vitest run src -t "packages/world-postgres/README.md" in packages/docs-typecheck — 3 samples pass, including the new one.
  • bun ./scripts/lint.ts in docs — 0 errors.
  • The three test/*.test.ts integration suites could not run: they start a Postgres testcontainer and there is no container runtime available here. They are untouched by this change, which adds no storage or SQL code.

Docs: role documented alongside queueConcurrency and applicationManagedShutdown in packages/world-postgres/README.md (options table, environment-variable table, and a short "Producer-only processes" section), in docs/content/docs/v5/configuration/worlds.mdx, and as WORKFLOW_POSTGRES_ROLE in docs/content/worlds/v5/postgres.mdx.

Closes #3968

PR Checklist - Required to merge

  • 📦 pnpm changeset was run to create a changelog for this PR
  • 🔒 DCO sign-off passes (run git commit --signoff on your commits)
  • 📝 Ping @vercel/workflow in a comment once the PR is ready, and the above checklist is complete

🤖 Generated with Claude Code

PostgresWorldRoleSchema is exported as a value from the package index alongside the PostgresWorldRole type, so a consumer can pass role: PostgresWorldRoleSchema.enum.producer instead of spelling the literal.

@Ehco1996
Ehco1996 requested a review from a team as a code owner September 4, 2026 09:38
@changeset-bot

changeset-bot Bot commented Sep 4, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 633b58d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@workflow/world-postgres Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

@Ehco1996 is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@Ehco1996
Ehco1996 force-pushed the feat/world-postgres-producer-role branch from 795f1c2 to 2a2c3cc Compare September 4, 2026 09:48
…r startup recovery

`queue()` awaits `start()`, and `start()` both starts a graphile-worker runner
and calls `reenqueueActiveRuns()`. A process that only ever enqueues therefore
gets both, and in the common "API process starts runs, worker process executes
them" topology both are unwanted: the API's runner claims due jobs and fails
their HTTP delivery whenever the worker has no ready endpoint, and every API
replica replays every active run on startup, including runs a live peer is
executing.

Add a `role` option to `PostgresWorldConfig`, its vocabulary declared once as a
zod enum in `config.ts`, defaulting to `worker` so existing behaviour is
unchanged. `createWorld()` resolves the role once — option, then
`WORKFLOW_POSTGRES_ROLE`, then the default — and hands it to `createQueue()`, so
nothing else reads the environment. Under `producer`, `start()` still creates
the worker utils, migrates the schema, and runs the pg-boss job migration — so a
producer can enqueue into a fresh database — but never calls graphile's `run()`
and never re-enqueues active runs.
`queue()` is untouched, and `close()` already tolerates a null runner, so it
ends only what was opened.

This removes N of the N+1 startup-recovery sources behind vercel#3119 and vercel#3758
without touching the recovery predicate itself.

Closes vercel#3968

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Ehco <ehco@mewtant.io>
@Ehco1996
Ehco1996 force-pushed the feat/world-postgres-producer-role branch from 2a2c3cc to 633b58d Compare September 5, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

world-postgres: a producer-only World — enqueue without a runner or startup recovery

1 participant