Skip to content

test: state the timeout budget for the two workflow suites that execute externally - #437

Merged
jpr5 merged 1 commit into
mainfrom
fix/external-test-timeouts
Sep 13, 2026
Merged

test: state the timeout budget for the two workflow suites that execute externally#437
jpr5 merged 1 commit into
mainfrom
fix/external-test-timeouts

Conversation

@jpr5

@jpr5 jpr5 commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

The failure

main @ 079f37e failed the test (24) leg of Unit Tests (run 34731065578):

FAIL src/__tests__/publish-pin-workflow.test.ts
 > the PyPI publish path runs no Python bytes it has not pinned
 > EXECUTED against real pip > SUBSTITUTED backend bytes are REFUSED by the repo's own pin
Error: Test timed out in 5000ms.        (file: 21075ms)

Not a correctness failure. The test shells out to real pip — deliberately, that is the guard — and inherited vitest's 5000ms default. It passes on a fast machine and fails on a slow one.

Audit

Every src/__tests__/*workflow*.test.ts and sibling that spawns a process was measured. Three already state a budget (fix-drift-workflow 30s, unreleased-check-workflow 30s, npm-publish-verify-workflow 60s). Two did not:

file slowest test, measured new budget
publish-pin-workflow.test.ts 172ms / 183ms warm local pip; 21075ms for the file on the CI runner 60_000
test-drift-workflow.test.ts 3039ms idle 18-core; 5490ms → timed out under concurrent load 30_000

drift-sync-core.test.ts also shells out to real git, but its slowest test is 116ms — 43x of headroom under the default. Left alone. fix-drift.test.ts mocks child_process. adoption-wall.test.ts is out of scope.

RED

test-drift-workflow at the inherited default, with two full suites running concurrently:

× EXECUTED POSITIVE CONTROL: base exits 0 and 2 stay non-fatal 5490ms
 Test Files  1 failed | 1 passed (2)
      Tests  1 failed | 32 passed (33)

publish-pin-workflow's RED is the CI run above — local pip is warm here and never approaches 5s.

GREEN

Both files, 5 consecutive runs:

run 1 exit=0    Tests  33 passed (33)
run 2 exit=0    Tests  33 passed (33)
run 3 exit=0    Tests  33 passed (33)
run 4 exit=0    Tests  33 passed (33)
run 5 exit=0    Tests  33 passed (33)

And once under the exact condition that produced RED — concurrent with two full suites:

CONCURRENT EXIT=0
 Test Files  2 passed (2)
      Tests  33 passed (33)
slowest under load: 4751ms, 4379ms, 3898ms

4751ms against a 5000ms budget is the flake, stated plainly.

The budgets still bite

A raised-so-far-it-can-never-fire timeout is a removed guard. Both were confirmed against an injected hang (sleep in front of the external call, reverted after):

publish-pin  → Test timed out in 60000ms.   Tests  2 failed | 7 passed (9)
test-drift   → Test timed out in 30000ms.   Tests  1 failed | 1 passed | 22 skipped (24)

Deliberately not done

  • No global testTimeout — that would stop genuinely hung tests from failing everywhere.
  • No stubbing. These are marked EXECUTED against real pip because running the repo's own pin file against the real resolver is the assertion. What they exercise is unchanged.
  • npm-publish-verify-workflow.test.ts untouched — already fixed in test(ci): drive the npm-verify retry loop on a virtual clock #436 on a stubbed clock.

Gates (raw exit codes)

gate exit
prettier --check 0
npx eslint . 0
pnpm typecheck 0
npx vitest run (full) 0 — 191 passed / 1 skipped, 5864 tests
npx commitlint --from origin/main --to HEAD 0

🤖 Generated with Claude Code

https://claude.ai/code/session_01AvkmhXVLqSSEW6FvPQHSu5

…te externally

`publish-pin-workflow.test.ts` and `test-drift-workflow.test.ts` inherited
vitest's 5000ms default while shelling out to real pip and running workflow
`run:` bodies under bash. Both were sized by the runner, not by the assertion.

publish-pin took the CI file to 21075ms on 2026-09-13 and failed `test (24)`
on main (run 34731065578); it measures ~0.2s per pip guard against a warm
local pip, so the default was never a real budget there. test-drift's slowest
EXECUTED guard measured 3039ms idle and 5490ms — a timeout — when re-measured
under concurrent load.

Budgets are stated per file, as fix-drift-workflow, unreleased-check-workflow
and npm-publish-verify-workflow already do, rather than raised globally: a
global raise would stop genuinely hung tests from failing anywhere. Both new
budgets were confirmed to still fire against an injected hang.

What these suites exercise is unchanged — the pip guards still run real pip.
@pkg-pr-new

pkg-pr-new Bot commented Sep 13, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@437

commit: 0b343d8

@jpr5
jpr5 merged commit 48e139e into main Sep 13, 2026
30 checks passed
@jpr5
jpr5 deleted the fix/external-test-timeouts branch September 13, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant