Release: survive a single failed publish, and verify every package reached npm - #3967
Release: survive a single failed publish, and verify every package reached npm#3967pranaygp wants to merge 2 commits into
Conversation
… actually reached npm Publishing 5.0.0-beta.48 shipped 7 of 21 packages on the first attempt and 5 more on a rerun; 4 are still not on npm. Two things compounded: - @changesets/cli 2.29.8 crashes on an E401 whose JSON has no `detail` field, which is the shape pnpm 11 produces when its OIDC token exchange fails and it falls back to the (empty) .npmrc token. The TypeError killed the process with other publishes mid-flight, and hid pnpm's own warning that says why OIDC was skipped. Patch the CLI (pnpm patchedDependencies) to treat a missing `detail` as a plain publish failure for that one package and to print pnpm's message, so the rest of the release proceeds and the log names the cause. - Nothing compared the manifests on the commit with the registry, so a half-published release looked like any other red job. Add scripts/check-published.mjs and run it after the publish step regardless of that step's outcome. Also stop a broken Slack token from failing an otherwise successful release; that is what turned the Aug 26 and Aug 31 release jobs red. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3636 | 0 | 684 | 4320 |
| ✅ 💻 Local Development | 2584 | 0 | 456 | 3040 |
| ✅ 📦 Local Production | 2718 | 0 | 482 | 3200 |
| ✅ 🐘 Local Postgres | 2718 | 0 | 482 | 3200 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ✅ 🌐 Cross-language Conformance | 68 | 0 | 73 | 141 |
| ✅ vercel-http-transport | 817 | 0 | 143 | 960 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 553 | 0 | 87 | 640 |
| Total | 13441 | 0 | 2407 | 15848 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 132 | 0 | 28 |
| ✅ astro-quickjs | 132 | 0 | 28 |
| ✅ example-node | 132 | 0 | 28 |
| ✅ example-quickjs | 132 | 0 | 28 |
| ✅ express-node | 132 | 0 | 28 |
| ✅ express-quickjs | 132 | 0 | 28 |
| ✅ fastify-node | 132 | 0 | 28 |
| ✅ fastify-quickjs | 132 | 0 | 28 |
| ✅ hono-node | 132 | 0 | 28 |
| ✅ hono-quickjs | 132 | 0 | 28 |
| ✅ nest-node | 132 | 0 | 28 |
| ✅ nest-quickjs | 132 | 0 | 28 |
| ✅ nextjs-turbopack-node | 157 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 157 | 0 | 3 |
| ✅ nextjs-webpack-node | 157 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 157 | 0 | 3 |
| ✅ nitro-node | 132 | 0 | 28 |
| ✅ nitro-quickjs | 132 | 0 | 28 |
| ✅ nuxt-node | 132 | 0 | 28 |
| ✅ nuxt-quickjs | 132 | 0 | 28 |
| ✅ python-node | 66 | 0 | 94 |
| ✅ sveltekit-node | 151 | 0 | 9 |
| ✅ sveltekit-quickjs | 151 | 0 | 9 |
| ✅ tanstack-start-node | 132 | 0 | 28 |
| ✅ tanstack-start-quickjs | 132 | 0 | 28 |
| ✅ vite-node | 132 | 0 | 28 |
| ✅ vite-quickjs | 132 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 68 | 0 | 73 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 132 | 0 | 28 |
| ✅ express | 132 | 0 | 28 |
| ✅ hono | 132 | 0 | 28 |
| ✅ nextjs-turbopack | 157 | 0 | 3 |
| ✅ nitro | 132 | 0 | 28 |
| ✅ vite | 132 | 0 | 28 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 132 | 0 | 28 |
| ✅ express | 132 | 0 | 28 |
| ✅ nextjs-turbopack | 157 | 0 | 3 |
| ✅ vite | 132 | 0 | 28 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 131526ms → this run 139152ms (Δ +7626ms, +6%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (1)f170303Fri, 04 Sep 2026 00:17:26 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core release/publishing behavior and should be validated by a maintainer with a real Release workflow run against npm before approval.
Pull request overview
Hardens the Release workflow so a single package publish failure doesn’t stop the rest of the publish loop, and so the workflow reliably reports whether the commit’s intended package versions actually reached npm (instead of failing “mysteriously” and leaving a half-shipped release unnoticed).
Changes:
- Add
scripts/check-published.mjsto cross-check each publishable package’s manifest version and expected dist-tag against the npm registry (with retries for registry lag). - Patch
@changesets/cli@2.29.8via pnpmpatchedDependenciesso missingjson.error.detailno longer crashes Changesets during publish error handling. - Make the Slack notification non-blocking (
continue-on-error: true) so notification failures don’t mask publish correctness.
File summaries
| File | Description |
|---|---|
| scripts/check-published.mjs | New post-publish verification script comparing repo manifests vs npm registry versions/dist-tags. |
| pnpm-workspace.yaml | Enables pnpm patchedDependencies for the Changesets CLI hotfix. |
| pnpm-lock.yaml | Lockfile updates to record the applied patch hash for @changesets/cli@2.29.8. |
| patches/@changesets__cli@2.29.8.patch | Patch preventing a publish-loop crash on missing error.detail and improving error logging fallback. |
| .github/workflows/release.yml | Runs check-published after Changesets with always(), and makes Slack step non-blocking. |
Review details
Files not reviewed (1)
- pnpm-lock.yaml: Generated file
- Files reviewed: 4/5 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| const url = `${REGISTRY}/${name.replace('/', '%2F')}`; | ||
| const response = await fetch(url, { | ||
| headers: { accept: 'application/json', 'cache-control': 'no-cache' }, | ||
| }); |
There was a problem hiding this comment.
A transient npm registry error (5xx/429/network/malformed JSON) thrown from registryState() propagates through check() and rejects the top-level Promise.all, crashing the check script and marking a fully-successful Release job red.
About these numbersSizes are gzip; parentheses show the change against
|
Follow-up to #3963 and the 5.0.0-beta.48 publish (run 33809804842).
What happened
The beta.48 release shipped 7 of 21 packages on attempt 1 and 5 more on a manual rerun. As of now 4 are still not on npm:
@workflow/nest,@workflow/nuxt,@workflow/sveltekit,@workflow/web(theirbetatag still points at beta.47).Two problems compounded:
@changesets/cli2.29.8 crashes on the error pnpm 11 produces. The workflow publishes with npm trusted publishing (no token,id-token: write). pnpm 11, which chore: upgrade to pnpm 11.24.0 #3901 moved us to on Sep 1, reimplements the OIDC exchange itself and, when it fails, only warns (Skipped OIDC: ...) and falls back to the.npmrctoken, which is empty here, so npm answers E401. Changesets then evaluatesjson.error.detail.includes(...)on an error object with nodetailand dies:libnpmpublish, which is why the Aug 26 and Aug 31 releases published cleanly.token_expired,invalid_auth), which trained everyone to treat a red Release job as noise.Changes
patches/@changesets__cli@2.29.8.patch(via pnpmpatchedDependencies): a missingdetailis treated as an ordinary failed publish for that one package, and the error line prints pnpm'smessageinstead ofundefined. The other packages keep publishing, and the log shows pnpm'sSkipped OIDC: Failed token exchange request with body message: ... (status code ...)warning so the next occurrence tells us the root cause. Upstream fixed this class in 3.0.0 ([core] Extend OCC fence to all branch-decision writes #2132), but that is a major with an unrelated migration (.changeset/pre/layout,pre.jsonremoval), so a two-line patch is the right size for now. The lockfile diff is the three entries pnpm needs for the patch; runningpnpm installinstead re-threads asupports-colorpeer through ~2,300 lines, so the entries were added by hand and validated withpnpm install --frozen-lockfile("Lockfile is up to date, resolution step is skipped").scripts/check-published.mjs: for every non-private package changesets does not ignore, checks that the manifest version is in the registry's version list and that the branch's dist-tag (the pre-release tag frompre.jsonin pre mode, otherwiselatest) points at it. Retries a few times for registry lag.release.ymlruns it after the publish step withalways(), so it reports even when the publish step crashes. The invariant holds on every commit of a release branch, so a gap keeps the job red until closed.continue-on-error: true. The token still needs rotating (it has failed since Aug 26), but that should not mask the publish result.Verified locally
Against the registry right now:
pnpm install --frozen-lockfilesucceeds and the installed@changesets/clidist contains the patched line.Finishing beta.48 after this lands
nuxtnever reached the registry: dispatch the Release workflow onmain;changeset publishskips the 22 live versions and publishes it (and now logs why if OIDC fails again).nest,sveltekit,webwere reported as previously staged/published but are not visible: a maintainer needs to approve them in each package's Staged Packages tab on npmjs.com (2FA), or reject them and rerun.pnpm changeset tagafter everything is visible will create the tags.No changeset: CI, scripts, and root tooling only.
🤖 Generated with Claude Code