Stress-test background hook subscribers and merged hook inboxes; add a cookbook recipe - #3986
Open
pranaygp wants to merge 9 commits into
Open
Stress-test background hook subscribers and merged hook inboxes; add a cookbook recipe#3986pranaygp wants to merge 9 commits into
pranaygp wants to merge 9 commits into
Conversation
…s and merged hook inboxes Two userland patterns an agent session needs from hooks: a `for await` over a hook that the workflow body never awaits, pushing payloads into a local inbox that each turn drains for steering; and several hooks merged into one async iterator, including a hook added to the merge after the run started (a Slack thread whose id is only known after the first reply). Both are plain JavaScript, so what the new e2e tests establish is that hook delivery stays in event-log order relative to step results: the inbox a turn drains on the live run is the inbox every replay drains at that turn, checked by echoing the drained messages back through the step's recorded arguments. Six tests cover a fixed-turn loop under paced bursts, a run that returns while its subscriber is still parked on the hook, a drain-then-wait session loop, a three-hook merge with round-robin sends (exact log order), concurrent senders with a step per message (per-hook order), and a hook added mid-run. `E2E_INBOX_SCALE` multiplies the message counts for soaks. The cookbook page (v5 only) ships the subscriber, the `mergeAsyncIterables` helper, and the `subscribeInbox` drain/wait wrapper, with a steering example and the Slack-thread shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
🦋 Changeset detectedLatest commit: bff32ad The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 131526ms → this run 124020ms (Δ -7506ms, -6%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (1)8fe31b3Fri, 04 Sep 2026 22:02:40 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests💻 Local Development (216 failed)express-stable-quickjs (21 failed):
fastify-stable-quickjs (19 failed):
hono-stable-quickjs (42 failed):
nest-stable-quickjs (50 failed):
nextjs-turbopack-canary-quickjs (25 failed):
nextjs-turbopack-stable-quickjs (37 failed):
nitro-stable-quickjs (22 failed):
📦 Local Production (263 failed)astro-stable-quickjs (57 failed):
express-stable-quickjs (50 failed):
nextjs-turbopack-stable-quickjs (53 failed):
nitro-stable-quickjs (58 failed):
sveltekit-stable-quickjs (45 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3729 | 0 | 834 | 4563 |
| ❌ 💻 Local Development | 2905 | 216 | 428 | 3549 |
| ❌ 📦 Local Production | 2584 | 263 | 364 | 3211 |
| ✅ 🐘 Local Postgres | 4174 | 0 | 558 | 4732 |
| ✅ 🪟 Windows | 169 | 0 | 0 | 169 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 82 | 150 |
| ✅ vercel-http-transport | 871 | 0 | 143 | 1014 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 589 | 0 | 87 | 676 |
| Total | 15116 | 479 | 2496 | 18091 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 141 | 0 | 28 |
| ✅ astro-quickjs | 141 | 0 | 28 |
| ✅ example-node | 141 | 0 | 28 |
| ✅ example-quickjs | 141 | 0 | 28 |
| ✅ express-node | 141 | 0 | 28 |
| ✅ express-quickjs | 141 | 0 | 28 |
| ✅ fastify-node | 141 | 0 | 28 |
| ✅ fastify-quickjs | 141 | 0 | 28 |
| ✅ hono-node | 141 | 0 | 28 |
| ✅ hono-quickjs | 141 | 0 | 28 |
| ✅ nest-node | 0 | 0 | 169 |
| ✅ nest-quickjs | 141 | 0 | 28 |
| ✅ nextjs-turbopack-node | 166 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 166 | 0 | 3 |
| ✅ nextjs-webpack-node | 166 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 166 | 0 | 3 |
| ✅ nitro-node | 141 | 0 | 28 |
| ✅ nitro-quickjs | 141 | 0 | 28 |
| ✅ nuxt-node | 141 | 0 | 28 |
| ✅ nuxt-quickjs | 141 | 0 | 28 |
| ✅ python-node | 66 | 0 | 103 |
| ✅ sveltekit-node | 160 | 0 | 9 |
| ✅ sveltekit-quickjs | 160 | 0 | 9 |
| ✅ tanstack-start-node | 141 | 0 | 28 |
| ✅ tanstack-start-quickjs | 141 | 0 | 28 |
| ✅ vite-node | 141 | 0 | 28 |
| ✅ vite-quickjs | 141 | 0 | 28 |
❌ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 143 | 0 | 26 |
| ✅ express-stable-node | 143 | 0 | 26 |
| ❌ express-stable-quickjs | 122 | 21 | 26 |
| ✅ fastify-stable-node | 143 | 0 | 26 |
| ❌ fastify-stable-quickjs | 124 | 19 | 26 |
| ✅ hono-stable-node | 143 | 0 | 26 |
| ❌ hono-stable-quickjs | 101 | 42 | 26 |
| ✅ nest-stable-node | 143 | 0 | 26 |
| ❌ nest-stable-quickjs | 93 | 50 | 26 |
| ✅ nextjs-turbopack-canary-node | 150 | 0 | 19 |
| ❌ nextjs-turbopack-canary-quickjs | 125 | 25 | 19 |
| ✅ nextjs-turbopack-stable-node | 169 | 0 | 0 |
| ❌ nextjs-turbopack-stable-quickjs | 132 | 37 | 0 |
| ✅ nextjs-webpack-canary-node | 150 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 169 | 0 | 0 |
| ✅ nitro-stable-node | 143 | 0 | 26 |
| ❌ nitro-stable-quickjs | 121 | 22 | 26 |
| ✅ nuxt-stable-node | 143 | 0 | 26 |
| ✅ sveltekit-stable-node | 162 | 0 | 7 |
| ✅ tanstack-start-node | 143 | 0 | 26 |
| ✅ vite-stable-node | 143 | 0 | 26 |
❌ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 143 | 0 | 26 |
| ❌ astro-stable-quickjs | 86 | 57 | 26 |
| ✅ express-stable-node | 143 | 0 | 26 |
| ❌ express-stable-quickjs | 93 | 50 | 26 |
| ✅ fastify-stable-node | 143 | 0 | 26 |
| ✅ hono-stable-node | 143 | 0 | 26 |
| ✅ nest-stable-node | 143 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 150 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 169 | 0 | 0 |
| ❌ nextjs-turbopack-stable-quickjs | 116 | 53 | 0 |
| ✅ nextjs-webpack-canary-node | 150 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 169 | 0 | 0 |
| ✅ nitro-stable-node | 143 | 0 | 26 |
| ❌ nitro-stable-quickjs | 85 | 58 | 26 |
| ✅ nuxt-stable-node | 143 | 0 | 26 |
| ✅ sveltekit-stable-node | 162 | 0 | 7 |
| ❌ sveltekit-stable-quickjs | 117 | 45 | 7 |
| ✅ tanstack-start-node | 143 | 0 | 26 |
| ✅ vite-stable-node | 143 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 143 | 0 | 26 |
| ✅ astro-stable-quickjs | 143 | 0 | 26 |
| ✅ express-stable-node | 143 | 0 | 26 |
| ✅ express-stable-quickjs | 143 | 0 | 26 |
| ✅ fastify-stable-node | 143 | 0 | 26 |
| ✅ fastify-stable-quickjs | 143 | 0 | 26 |
| ✅ hono-stable-node | 143 | 0 | 26 |
| ✅ hono-stable-quickjs | 143 | 0 | 26 |
| ✅ nest-stable-node | 143 | 0 | 26 |
| ✅ nest-stable-quickjs | 143 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 150 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 150 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 169 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 169 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 150 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 150 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 169 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 169 | 0 | 0 |
| ✅ nitro-stable-node | 143 | 0 | 26 |
| ✅ nitro-stable-quickjs | 143 | 0 | 26 |
| ✅ nuxt-stable-node | 143 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 143 | 0 | 26 |
| ✅ sveltekit-stable-node | 162 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 162 | 0 | 7 |
| ✅ tanstack-start-node | 143 | 0 | 26 |
| ✅ tanstack-start-quickjs | 143 | 0 | 26 |
| ✅ vite-stable-node | 143 | 0 | 26 |
| ✅ vite-stable-quickjs | 143 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 169 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 82 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 141 | 0 | 28 |
| ✅ express | 141 | 0 | 28 |
| ✅ hono | 141 | 0 | 28 |
| ✅ nextjs-turbopack | 166 | 0 | 3 |
| ✅ nitro | 141 | 0 | 28 |
| ✅ vite | 141 | 0 | 28 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 141 | 0 | 28 |
| ✅ express | 141 | 0 | 28 |
| ✅ nextjs-turbopack | 166 | 0 | 3 |
| ✅ vite | 141 | 0 | 28 |
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new subscribeInbox helper (and its cookbook snippet) can hang in wait() when the source iterable ends normally, and the cookbook index link path is inconsistent with existing cookbook links.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds stress-test coverage for two common “hook inbox” patterns used by agent-style workflows (background subscribers and merged inboxes), and documents the patterns as a new v5 cookbook recipe.
Changes:
- Add new e2e workflows demonstrating background hook subscribers, drain/wait inbox loops, and merged async-iterator inboxes (including dynamic hook addition mid-run).
- Add a new
hook inbox patternse2e test suite that validates determinism via event-log ordering and asserts expected event counts. - Add a v5-only cookbook recipe entry and page documenting the patterns, plus a Changeset for
@workflow/core.
File summaries
| File | Description |
|---|---|
| workbench/example/workflows/99_e2e.ts | Adds new workflows and helpers implementing background inbox + merged-hook inbox patterns for e2e. |
| packages/core/e2e/e2e.test.ts | Adds an e2e suite that stresses and validates inbox determinism and event-log integrity under load. |
| docs/lib/cookbook-tree.ts | Registers the new cookbook recipe slug in the cookbook navigation data. |
| docs/content/docs/v5/cookbook/index.mdx | Adds the new recipe to the cookbook index list. |
| docs/content/docs/v5/cookbook/agent-patterns/meta.json | Adds hook-inbox to the agent-patterns page list. |
| docs/content/docs/v5/cookbook/agent-patterns/hook-inbox.mdx | Introduces the new “Hook Inbox & Steering” recipe (v5-only). |
| .changeset/hook-inbox-e2e.md | Declares a patch bump for @workflow/core for the added e2e coverage. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+4080
to
+4090
| const done = (async () => { | ||
| for await (const value of source) { | ||
| if (isEnd(value)) { | ||
| ended = true; | ||
| bump(); | ||
| break; | ||
| } | ||
| buffered.push(value); | ||
| bump(); | ||
| } | ||
| })(); |
Contributor
Author
There was a problem hiding this comment.
Good catch, fixed in 1b6cd3a: the subscriber now sets ended and wakes waiters in a finally, so wait() unblocks whether the source hit the end marker, completed on its own, or threw. Same fix applied to the cookbook snippet, which now lives on the new Background Hook Subscriber page.
Comment on lines
+218
to
+228
| const done = (async () => { | ||
| for await (const value of source) { | ||
| if (isEnd(value)) { | ||
| ended = true; | ||
| bump(); | ||
| break; | ||
| } | ||
| buffered.push(value); | ||
| bump(); | ||
| } | ||
| })(); |
Contributor
Author
There was a problem hiding this comment.
Good catch, fixed in 1b6cd3a: the subscriber now sets ended and wakes waiters in a finally, so wait() unblocks whether the source hit the end marker, completed on its own, or threw. Same fix applied to the cookbook snippet, which now lives on the new Background Hook Subscriber page.
Comment on lines
+11
to
+14
| - [**WorkflowAgent**](/cookbook/agent-patterns/durable-agent): Build durable, resumable AI agents with AI SDK's WorkflowAgent | ||
| - [**Human-in-the-Loop**](/cookbook/agent-patterns/human-in-the-loop): Pause an agent for human approval, then resume based on the decision | ||
| - [**Agent Cancellation**](/cookbook/agent-patterns/agent-cancellation): Stop a running agent immediately via `run.cancel()` or gracefully via a hook + `Promise.race` | ||
| - [**Hook Inbox & Steering**](/docs/cookbook/agent-patterns/hook-inbox): Buffer hook payloads in a background subscriber to steer an agent loop, and merge several hooks into one ordered inbox |
Contributor
Author
There was a problem hiding this comment.
Deliberate, and the docs link lint requires it. In docs/lib/geistdocs/source.ts the current docs version is v4, so a bare /cookbook/... href resolves to the v4 cookbook, where this v5-only page does not exist (it is registered with skipVersions: ["v4"]). /docs/cookbook/... is rewritten at render time to /v5/cookbook/... on v5 pages (rewriteLocalDocsUrlForVersion), which is the page that exists. With the bare prefix bun docs/scripts/lint.ts reports the link as not-found; with this prefix it passes. The other entries can use the bare prefix because those recipes exist in both versions.
Contributor
About these numbersSizes are gzip; parentheses show the change against
|
…ding the wait loop On every Vercel lane the early-return test failed with HookNotFoundError: each resume is slow enough there that the run finished its four turns and `using` disposed the hook while the test was still sending. The workflow now takes a `minMessages` floor and keeps turning until its subscriber has seen that many payloads, which is as replay-safe as reading the inbox. The drain-then-wait loop checked `ended` before draining, so payloads pushed between the last drain and the end marker were left over; it cost one retry on some Vercel lanes. The workflow and the cookbook example now drain first. The dynamic-inbox test's thread-hook lookup also gets the 120s budget the event waits use, since it depends on a step completing first. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`// @setup` marks type-check-only declarations in docs code samples (#846), originally hidden client-side by a custom CodeBlock that the geistdocs migration (#2222) dropped, so every marked line has rendered since. A remark plugin now removes them once at build time, whole multi-line declarations included, covering the rendered page and the processed-markdown exports. The Background Hook Subscriber recipe gains the hand-off ending: when the loop exits on its own, release the hook, commit the release with a step, drain once more, and start a successor run with anything left. The new `handoffInboxWorkflow` e2e test drives it with a sender that runs until the parent completes and follows the successor chain. That test found a runtime gap the pattern cannot close: `resumeHook` is accepted until `hook_disposed` commits at the next suspension, but `hook.dispose()` stops the in-memory iterator immediately, so a payload that lands in between is buffered with no consumer and lost (2 of 7 postgres runs, 3 of 13 world-local runs, always exactly one payload sitting between the last turn's `step_completed` and `hook_disposed`). The test asserts today's guarantee and bounds the loss by counting those log positions; the docs say so in a callout. Closing it means `dispose()` keeping delivery of payloads that precede the disposal event, which changes its documented semantics and is left as a proposal. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…gence On Vercel the return value is polled every 5s, so the sender outlives the parent and the successor hands off again; the test now follows the chain instead of asserting two generations. The lanes also showed the disposal window from the other side: the parent returned late=[2,3,4] while its successor was started with [2,3], because the successor's arguments were recorded on the live retained-VM run and the return value came from a fresh replay that did deliver the window payload. The test asserts the carried set is a prefix of the parent's late and bounds the excess, like the drops, by the window payloads in each generation's log. The docs callout says late is not an exact record of what the successor received until dispose() keeps delivering pre-disposal payloads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…engines `resumeHook` is accepted until `hook_disposed` commits at the run's next suspension, but `hook.dispose()` stopped the in-memory iterator immediately. A payload landing in between sat in the log ahead of the disposal with no consumer left to claim it: lost on the retained-VM path, delivered by a fresh replay, so the workflow's view was replay-inconsistent too. The hand-off e2e test hit both (2/7 postgres, 3/13 world-local runs; `late=[2,3,4]` returned against a successor started with `[2,3]` on Vercel). Disposal is now the durable event it always was on the world side. After `dispose()`, awaiters and the `for await` iterator keep receiving payloads that precede `hook_disposed`, and iteration ends when that event is observed. node:vm: iterator waiters are tracked with their payload resolvers so the disposal event ends only the idle ones (a waiter already receiving a payload ends on its next pull), settled through promiseQueue for log order; `disposeHook` no longer drops awaiters or schedules a suspension, since a waiter parked at end of log already does. QuickJS: the host marks the hook and resolves a parked iterator with a sentinel when it processes `hook_disposed`; `next()` returns done only once the buffer is drained and the flag is set, and a plain await stays pending on the sentinel. Unit test pins the shape; the hand-off e2e test asserts zero loss and `late == carried` again and logs how many payloads fell in the window. dispose() docs updated in the JSDoc, API reference, foundations page, and the cookbook recipe. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e; add ordering stress tests Two new e2e tests probe the merged inbox where the round-robin test could not: a seeded, bursty cross-hook send order (A,A,A,B,C,C,A,...) delivered while each merged item costs a 400 ms step, so payloads buffer across hooks and a replay must order them at claim time, plus a hook that only ever receives its done and another that closes early. Both compare the replay's merge order with the arguments the live run recorded per step. They failed against the Promise.race helper: replay-deterministic, but draining one hook at a time. Two causes, both in the helper. A race over already-settled promises picks by source order, and pulling a source only when its item is consumed means a payload buffered for hook A cannot be ordered against hook B's until A's previous item is yielded, which degrades to round-robin. The helper now records each source's next() result in a FIFO as it resolves and keeps exactly one next() outstanding per live source, so the runtime's delivery barriers, which resolve pending hook awaits in log order, define the merged order. Same change in the cookbook, with the reasoning. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validates two userland hook patterns an agent session needs, adds e2e coverage that stresses them on every world, and documents them as two cookbook recipes.
for await (const m of hook)loop that the workflow body never awaits, pushing payloads into a local inbox that each turn drains for steering.mergeAsyncIterableshelper, including a hook added to the merge after the run started (a Slack thread whose id is only known after the first reply).Both are ordinary JavaScript. What the tests establish is that hook delivery stays in event-log order relative to step results, so the inbox a turn drains on the live run is the inbox every replay drains at that turn. Each turn's drained messages are passed as the step's arguments and echoed back, so a replay that grouped payloads differently would disagree with the recorded arguments and fail the test. The event log is also checked for exact
hook_received/step_completedcounts and zerohook_conflict.Tests (
hook inbox patternsinpackages/core/e2e/e2e.test.ts, workflows inworkbench/example/workflows/99_e2e.ts)backgroundInboxWorkflowsteeringstep_completedso payloads land mid-stepbackgroundInboxWorkflowearly returnawait hook;usingreleases the tokeninboxWaitLoopWorkflowwait()session loop, each turn drains what arrived since the lastmergedHooksWorkfloworderedmergedHooksWorkflowconcurrentdynamicInboxWorkflowhandoffInboxWorkflowmergedHooksReplayCheckWorkflowirregularA,A,A,B,C,C,A,..., seed printed on failure), sent while each merged item costs a 400 ms step so payloads buffer mid-step; asserts exact log order and that the final replay's merge order equals the arguments the live run recorded per stepmergedHooksReplayCheckWorkflowsilent + early doneE2E_INBOX_SCALE=Nmultiplies every count for soaks.Local results
next start): 6/6 green on 8 consecutive runs at the default scale.E2E_INBOX_SCALE=4(480 payloads through the merged inbox, 300 through the step-per-message variant).E2E_INBOX_SCALE=4first OOMed the singlenext startprocess (heap limit) before any test finished: the known single-process saturation fromWORKFLOW_LOCAL_QUEUE_CONCURRENCYdefaulting to 1000 (see CLAUDE.md). Rerun withWORKFLOW_LOCAL_QUEUE_CONCURRENCY=10and an 8 GB heap: 5/6 green with every ordering assertion intact (the merged inbox reproduced the exact send order of 480 payloads), and the sixth,dynamicInboxWorkflow, failed only because its thread-hook lookup hitwaitForHook's 30 s default while six runs contended for ten queue slots. Alone at 4x it passes in 9 s, so that lookup now gets the same 120 s budget as the event waits. No ordering or replay-divergence failure was observed on any world.Vercel lanes (first CI run, commit 74bf7e3)
Every Vercel prod / WS / HTTP lane failed on exactly one test,
backgroundInboxWorkflow - run completes while the subscriber is still awaiting the hook, withHookNotFoundErrorfromresumeHook; the other five inbox tests passed on all of them (nextjs-webpack: 162 passed). The cause was the test, not the runtime: on Vercel each resume is slow enough that the run finished its four turns andusingdisposed the hook while the test was still sending. Fixed by giving the workflow aminMessagesfloor so it keeps turning until it has observed every message the test sends. The wait-loop test also needed one retry on some lanes because the loop checkedendedbefore draining, leaving payloads pushed between the last drain and the end marker unprocessed; the loop (and the cookbook example) now drain first. The unrelated failures in that run wereRetryableError respects custom retryAfter delay(Windows) andhookDisposeTestWorkflow(express dev quickjs), both pre-existing.Docs
Two cookbook recipes (v5 only, since the ordering guarantees were validated on v5):
subscribeInboxdrain/wait()wrapper (withwait()suspending durably), how a subscription ends, and why buffer reads after an await are replay-safe.mergeAsyncIterableshelper, and adding a hook to the merge mid-session (the Slack-thread shape).Also documented: the hand-off ending for a subscriber whose loop exits on its own. Release the hook, commit the release with a step, drain once more, and start a successor run with anything that beat the release, so nothing is dropped and a sender that lands after the release gets
HookNotFoundErrorrather than silence. Covered by the newhandoffInboxWorkflowe2e test, which sends continuously until the parent completes and follows the chain of successor runs.Runtime fix:
dispose()no longer orphans acknowledged payloads. The hand-off test found this: in 2 of 7 postgres runs and 3 of 13 world-local runs one acknowledged payload was processed by no generation, and on the Vercel lanes the parent returnedlate = [2,3,4]while its successor had been started with[2,3]. The parent's event log showed the cause each time: ahook_receivedafter the last turn'sstep_completedand beforehook_disposed.resumeHookis accepted untilhook_disposedcommits at the next suspension, buthook.dispose()stopped the in-memory iterator immediately, so a payload landing in between was buffered with no consumer: lost on the retained-VM path, delivered by a fresh replay, hence the divergence. No ordering of drain and dispose in workflow code can close this.Both engines now treat disposal as the durable event it is. After
dispose(), awaiters and thefor awaititerator keep receiving payloads that precedehook_disposedin the log, and iteration ends when the disposal event is observed (node:workflow/hook.tstracks iterator waiters with their payload resolvers so the disposal event ends only the idle ones, ordered throughpromiseQueue; QuickJS: the host marks the hook and resolves a parked iterator with a sentinel when it processeshook_disposed).dispose()no longer schedules its own suspension; a waiter still parked at end of log already does. A unit test pins the shape (payloads beforehook_disposed,dispose()called before any was consumed, iterator yields both then ends), and the hand-off e2e test is back to asserting zero loss andlate == carriedacross the chain, logging how many payloads fell in the window so runs that exercised it are recognizable.dispose()docs in the API reference, foundations page, and JSDoc are updated.Finding from the new merge tests:
Promise.raceis the wrong merge primitive here. Both new tests failed on both worlds on the first run with the same shape:recorded == observedheld everywhere (the merge is replay-deterministic), but the order wasA0..A11 B0..B11 C0..C11against a bursty interleaved send order. The runtime resolves the buffered claims in log order, but while the consumer is away in a step they all settle, andPromise.raceover already-settled promises picks by iteration order of the sources. The round-robin test never saw this because its payloads always found a waiting consumer. The helper (e2e and cookbook) now pushes each source'snext()result into a FIFO when it resolves and yields from that queue, which yields log order; the cookbook says why.QuickJS on world-local: the inbox tests are skipped there, and only there. The first CI run in which those lanes completed (every earlier push had cancelled them) failed 28 of them with mass timeouts, and I reproduced it locally: on world-local every
resumeHook()leaves a queue wake-up, delivered to the single app process with no concurrency bound and no check that the run is still running; on QuickJS each wake-up instantiates a fresh WASM VM and replays the whole log before finding the run completed (one 120-payload run was replayed 114 more times after finishing). These tests send 20 to 120 resumes per run. Concurrently that starves the process (ReplayTimeoutRetryErrorat 240 s, 2,804 flow-routefetch failedin the express lane's server log, every later test timing out); one at a time the nine tests took 61 minutes and still failed five. The same tests pass on QuickJS against postgres (bounded worker pool) and Vercel (one invocation per replay), so the engine is covered and the skip is scoped to that dev-only pairing at this resume rate. Follow-up worth doing separately: have the flow handler short-circuit wake-ups for runs that are already completed, which removes the amplification for every world.Docs pipeline fix.
// @setuplines (type-check-only declarations in code samples, introduced in #846 and hidden client-side by a customCodeBlock) have rendered on every page since the geistdocs migration (#2222) dropped that component; the production batching page shows them today.docs/lib/remark-strip-setup-lines.tsrestores the behavior as a remark plugin wired throughdefineGeistdocsSourceConfig, so the strip happens once at build time and covers the rendered page and the processed-markdown exports. It removes the whole statement when the marker sits on the last line of a multi-line declaration, then trims the blank lines it leaves. Verified on a localnext dev: both new pages render with zero@setupordeclaretokens.Docs typecheck and link lint pass. Review feedback addressed in 1b6cd3a:
subscribeInboxnow marks itself ended in afinally, sowait()cannot hang if the source completes or throws without an end marker.Docs Preview
The preview sits behind deployment protection, so the links need Vercel team access.
🤖 Generated with Claude Code