feat(control-api): journal filters, fixtures dump, and runtime chaos control - #435
Merged
Merged
Conversation
commit: |
jpr5
added a commit
that referenced
this pull request
Sep 13, 2026
## Why The `EXECUTED: the retry window is honoured, not multiplied by a longer deadline` negative control in `src/__tests__/npm-publish-verify-workflow.test.ts` asserted on **real elapsed wall clock**. It is red on `main` @ `252a4cc` (run 34597377003, legs `test (20)` and `test (26)`), which puts a red X on every open PR — including the three external-contributor PRs #433/#434/#435. Widening the window was already tried (`20e0d88`) and is not the fix. The step's body computes `ELAPSED` from `date +%s`, which is whole-second: if `START` is sampled just before a second boundary, the loop's own arithmetic crosses a 3s deadline after as little as ~2.0s of real time. That is sub-second phase, not runner load — it fails on an idle laptop. ## RED On `origin/main`, unloaded, 25 consecutive runs of just that test: **8 failed**. ``` RUN 6: RED → expected 2469 to be greater than or equal to 2800 RUN 7: RED → expected 2501 to be greater than or equal to 2800 RUN 8: RED → expected 2492 to be greater than or equal to 2800 RUN 9: RED → expected 2475 to be greater than or equal to 2800 RUN 16: RED → expected 2474 to be greater than or equal to 2800 RUN 17: RED → expected 2539 to be greater than or equal to 2800 RUN 21: RED → expected 2567 to be greater than or equal to 2800 RUN 22: RED → expected 2466 to be greater than or equal to 2800 ``` ## Fix `date` and `sleep` are external commands, so they can be stubbed on `PATH` exactly like the existing `npm` stub. The step's `run:` body is still executed **verbatim and unmodified** — only its clock is now the test's. - `sleep N` records `N` and advances a virtual clock file by `N`; `date +%s` reads it. - Timing becomes an **observable**: the exact attempt count and the exact backoff schedule the loop issued. No assertion in the file touches wall clock any more (`elapsedMs` is gone from `Observation`). - The observations now run at the **shipped** `300s`/`5s`/`30s` defaults instead of a shrunken 1s window, so what is tested is what CI runs. A five-minute window costs nothing on a virtual clock — the whole file got *faster*, ~30s → ~8s. - The default schedule is asserted exactly: attempts at `t+0,5,15,35,65,95,125,155,185,215,245,275,305`, sleeps `[5,10,20,30,30,30,30,30,30,30,30,30]`, failing at attempt 13. - "Honoured, not multiplied" is now a property that no runner speed can break: a shorter deadline stops the loop sooner, and overshoot past any deadline is `< deadline + MAX_DELAY` (one backoff step). - A body that stopped sleeping would spin against a clock that never advances, so the `npm` stub jumps the clock after 40 attempts — turning a hang into a legible failure. ## GREEN 20 consecutive green runs of the file: 12 standalone, then 8 more **concurrently with the full `npx vitest run`** (189 files / 5804 tests, exit 0) hammering the same machine. Zero failures. ## Mutation proof — it still bites Both mutations applied to the real `publish-release.yml` guard: | Mutation | Result | |---|---| | Retry loop deleted (`while :; do` → single pass, i.e. the pre-fix "ask once" body) | **RED — 5 of 10 tests fail** (`expected 1 to be 2`, `expected 1 to be 3`, `expected +0 to be 1` ×3) | | Deadline's `exit 1` → `exit 0` | **RED — 3 of 10 tests fail** (`expected +0 to be 1` ×3) | Workflow restored afterwards; this PR changes **one file**, the test. ## Other wall-clock assertions in the file Yes — three more, all fixed the same way: the `never appears FAILS` control (`elapsedMs >= 4000`), the `FAILS FAST` control (`elapsedMs < 10_000`), and the loose `attempts > N` bounds. All are now exact attempt counts and exact sleep schedules. ## Gates (raw exit codes) | Gate | Exit | |---|---| | `prettier --check` | 0 | | `npx eslint .` | 0 | | `pnpm typecheck` | 0 | | `npx vitest run` (full) | 0 | | `npx commitlint --from origin/main --to HEAD` | 0 | | `npx actionlint` | 1 — **pre-existing on `main`**, SC2086 in `changelog-radar.yml` / `test-drift.yml`; no workflow file is touched by this PR | 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AvkmhXVLqSSEW6FvPQHSu5
jpr5
force-pushed
the
feat/control-api-inspection
branch
from
September 13, 2026 01:51
fbb4fd6 to
dbb93a4
Compare
…ix dump kinds Follow-up fixes on this PR's new control surface, before it becomes a permanent public contract. - Runtime chaos is now scoped to the caller's X-Test-Id, like every other mutable axis in the server (fixture match-counts, video job maps). A global switch is unusable under the topology aimock is built for: one shared process serving a parallel suite, where one test enabling chaos failed every test running beside it. Scoping is by header on both sides -- the control call and the traffic it affects -- so the two can never resolve to different scopes. - POST /__aimock/reset now drops every chaos override, restoring the config the server was STARTED with. Reset is the isolation barrier a parallel harness leans on; chaos escaping it produced 500s that look like application bugs. This also fixes the in-process LLMock.reset(). - The override no longer latches: a server started with --chaos-drop was permanently destroyed by the first empty body, with no way back. Added DELETE /__aimock/chaos to drop one override without a full reset. - PUT -> POST. PUT would have been the only one in the whole control surface (everything else is GET/POST/DELETE) and it was missing from Access-Control-Allow-Methods, so the preflight blocked browser harnesses. POST needs no new CORS method. - Unknown journal query params are now 400 instead of silently returning the whole journal -- matching the 'include' validation this PR already had two routes down. ?statusCode=404 used to pass vacuously forever. - The testId filter no longer prefix-collides (?testId=t1 matched t10); it resolves each entry the way the server resolves a request. - fixtureResponseKind uses the existing is*Response guards instead of Object.keys()[0], which reported 'id' for any recorded fixture and ignored the audio-first ORDERING CONTRACT in types.ts. - The dump field holding a kind string is renamed responseKind; on Fixture, 'response' already means the response itself. - The journal's post-filter, pre-pagination total ships as X-Total-Count (exposed via CORS). The body stays a bare array for back-compat, so a total could never be added to it later.
… routes The control API page is hand-authored and is the only doc surface for these routes, so an undocumented endpoint is permanently undiscoverable. Adds the journal's query-param table and X-Total-Count, the ?include=fixtures dump, and the GET/POST/DELETE chaos routes with the per-test scoping rule. Rendered and checked at 1440x900, 375x667 and 430x932; page scroll width at 375 is unchanged from before this PR.
…leaking Five defects in the chaos/inspection control surface, each reproduced against a running server before the fix and re-run after. - `LLMock.setChaos()`/`clearChaos()` silently no-opped once an untagged `POST /__aimock/chaos` existed: the getter preferred the runtime override forever, shadowing `serverOptions.chaos`. They now write through to a running server's baseline slot, so the documented in-process API is not dead until the next `reset()`. - `X-Test-Id: ""` on a control call — trivially produced by `String(testId ?? "")` — took the untagged branch and installed a SERVER-WIDE baseline that failed every other test. A present-but-blank tag is now a 400 on GET, POST and DELETE. - An untagged `DELETE /__aimock/chaos` wiped every per-testId override, so one test's cleanup revoked a concurrently-running test's opt-out. Untagged DELETE is now symmetric with untagged POST: it drops the baseline only. The baseline lives in the same map under DEFAULT_TEST_ID, so "drop the baseline" and "drop one test's override" are the same operation and neither can take the other's state with it. `POST /__aimock/reset` remains the one full clear. - Chaos scoping read the `X-Test-Id` header only, while `getTestId` and the journal filter also honour `?testId=`: a harness tagging by query param installed chaos and watched it do nothing. Both sides now resolve through one exported `resolveTestId`, and `applyChaos` takes the raw `req.url` as a REQUIRED argument so a new handler cannot forget it. - `GET /__aimock/fixtures` silently ignored unknown query params while `/journal` 400s on them; `?incluide=fixtures` quietly returned the count-only body. It now 400s on any param but `include`.
Match the shipped behaviour: chaos scoping resolves `X-Test-Id` then `?testId=` on both the control call and the traffic; a present-but-empty `X-Test-Id` is a 400; an untagged DELETE drops the baseline only and leaves per-test overrides in place; `GET /__aimock/fixtures` rejects unknown query params.
jpr5
force-pushed
the
feat/control-api-inspection
branch
from
September 13, 2026 02:11
dbb93a4 to
d037eb3
Compare
…der only `ChaosScope`'s doc comment and the CHANGELOG entry both said scoping was by the `X-Test-Id` HEADER only on both sides. This PR's own fix made chaos resolve its scope through the shared `resolveTestId` — the header, else `?testId=` in the query string — used by `getTestId`, `chaosScopeId` and `resolveScopedDefaults` alike. Both statements now say what the code does and match the rendered control-api docs page, which was already correct. Text only; no behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
/__aimock/*control API is documented as the way for harnesses to manage fixtures/journal/chaos without restarting the server, but it was missing the read side:GET /journaldumped the entire unbounded journal with no filtering,GET /fixturesreturned only a count, and chaos rates were settable only via CLI flags at startup. All three gaps are closed here — strictly additive, existing responses unchanged when no new params are used:GET /__aimock/journal?limit=&offset=&path=&method=&status=&service=&testId=— filter/paginate the journal (e.g.?path=/search&status=200&limit=20). Invalidlimit/offset/status→ 400. No params → full array, exactly as before.GET /__aimock/fixtures?include=fixtures—{ count, fixtures: [{ index, match, response, latency?, chaos? }] }with functions redacted to"[function]"and RegExps stringified, so output is always JSON-safe. BareGETstill returns{ count }; unknownincludevalues → 400.GET /__aimock/chaos+PUT /__aimock/chaos— read/replace server-level chaos at runtime ({dropRate, malformedRate, disconnectRate}, each 0–1;{}clears). Unknown fields and out-of-range values → 400. Backed by a new override-awaredefaults.chaosgetter/setter that still falls through to construction options until PUT is used. Documented:POST /resetdoes not clear runtime chaos (PUT{}does).Verification
src/__tests__/control-inspection.test.ts: 12 tests — pagination, every filter, 400 matrix, dump redaction incl. predicate/RegExp fixtures, chaos read/apply/drop-then-clear cycle, chaos 400 matrix with nothing-applied assertion.control-api(23),chaos(36),chaos-fixture-mode(3),journal(40),server(64),llmock(91).pnpm typecheck,eslint,prettierclean.