test: state the timeout budget for the two workflow suites that execute externally - #437
Merged
Conversation
…te externally `publish-pin-workflow.test.ts` and `test-drift-workflow.test.ts` inherited vitest's 5000ms default while shelling out to real pip and running workflow `run:` bodies under bash. Both were sized by the runner, not by the assertion. publish-pin took the CI file to 21075ms on 2026-09-13 and failed `test (24)` on main (run 34731065578); it measures ~0.2s per pip guard against a warm local pip, so the default was never a real budget there. test-drift's slowest EXECUTED guard measured 3039ms idle and 5490ms — a timeout — when re-measured under concurrent load. Budgets are stated per file, as fix-drift-workflow, unreleased-check-workflow and npm-publish-verify-workflow already do, rather than raised globally: a global raise would stop genuinely hung tests from failing anywhere. Both new budgets were confirmed to still fire against an injected hang. What these suites exercise is unchanged — the pip guards still run real pip.
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
main@079f37efailed thetest (24)leg of Unit Tests (run 34731065578):Not a correctness failure. The test shells out to real pip — deliberately, that is the guard — and inherited vitest's 5000ms default. It passes on a fast machine and fails on a slow one.
Audit
Every
src/__tests__/*workflow*.test.tsand sibling that spawns a process was measured. Three already state a budget (fix-drift-workflow30s,unreleased-check-workflow30s,npm-publish-verify-workflow60s). Two did not:publish-pin-workflow.test.ts60_000test-drift-workflow.test.ts30_000drift-sync-core.test.tsalso shells out to realgit, but its slowest test is 116ms — 43x of headroom under the default. Left alone.fix-drift.test.tsmockschild_process.adoption-wall.test.tsis out of scope.RED
test-drift-workflowat the inherited default, with two full suites running concurrently:publish-pin-workflow's RED is the CI run above — local pip is warm here and never approaches 5s.GREEN
Both files, 5 consecutive runs:
And once under the exact condition that produced RED — concurrent with two full suites:
4751ms against a 5000ms budget is the flake, stated plainly.
The budgets still bite
A raised-so-far-it-can-never-fire timeout is a removed guard. Both were confirmed against an injected hang (
sleepin front of the external call, reverted after):Deliberately not done
testTimeout— that would stop genuinely hung tests from failing everywhere.EXECUTED against real pipbecause running the repo's own pin file against the real resolver is the assertion. What they exercise is unchanged.npm-publish-verify-workflow.test.tsuntouched — already fixed in test(ci): drive the npm-verify retry loop on a virtual clock #436 on a stubbed clock.Gates (raw exit codes)
prettier --checknpx eslint .pnpm typechecknpx vitest run(full)npx commitlint --from origin/main --to HEAD🤖 Generated with Claude Code
https://claude.ai/code/session_01AvkmhXVLqSSEW6FvPQHSu5