Skip to content

th-8e3087: SmoothFlow engine e2e — real daemon, fake-agent manifests, hooks contract, th flow - #562

Open
brentrager wants to merge 7 commits into
mainfrom
th-8e3087-e2e
Open

th-8e3087: SmoothFlow engine e2e — real daemon, fake-agent manifests, hooks contract, th flow#562
brentrager wants to merge 7 commits into
mainfrom
th-8e3087-e2e

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Problem

Brent: "can we write similar tests to orca / cmux". Lane A of th-6ac036 had engine unit tests, the macOS app its XCTests, the phones codec/reducer tests — nothing booted the real daemon and drove it the way a client, th flow or a harness's hook script does.

Solution

crates/smooth-daemon/tests/flow_e2e — a REAL smooth-daemon per test (isolated HOME, ephemeral port, private tmux server, no lock / tailscale / relay / gateway creds), driven over the flow WS, the HTTP siblings, the th binary and POST /api/flow/hooks. The agent under test is tests/fixtures/fake-agent, installed through harness manifests like any coding CLI, in four flavours — hooks / learned id / native / scrape — one per state source the engine supports.

24 tests, ~70 s wall (2 slow ones wait real backoff / usage-limit windows):

  • shell lifecycle: new → attach → input echo → resize → snapshot → kill; exit 0 = done, exit 7 = dead
  • agent transitions working / idle / needs_you / limited / done / dead; steering; the permission long-poll (all three decision shapes); usage-limit scheduling + the resume firing
  • resume-on-death with --resume <id>; the three-resume give-up; learned-id binding from the first hook; the duplicate-resume guard (held)
  • native (th code shape) and scrape (no hooks) state sources
  • the hooks contract event by event; hooks unauthenticated, everything else gated
  • th flow ls/new/snapshot/send/inbox/approve/handoff/kill --json + human lines; th harness list/add/hide/unhide/order/show
  • the per-harness state-source matrix; sort/hide prefs reaching flow.hello, the WS broadcast and th harness list; th harness add <path> of a custom manifest then a session on it; real claude/opencode/codex/th-code launches (opt-in, SMOOTH_E2E_REAL_HARNESSES=1)
  • proof the suite never touches the real ~/.smooth (daemon.addr, daemon.lock, operator-token, flow.db byte- and mtime-identical), teardown leaves nothing, and flow_e2e_server (the mac lane's host) is the same engine

Engine bugs the suite found on day one (fixed here, each with a unit test)

  1. tmux rewrites control characters under a non-UTF-8 locale — the tab-joined #{pane_dead}\t#{pane_dead_status} query read 1_2, so a dead pane was NEVER detected for any daemon not started from a shell (launchd, CI). Every supervision rule was inert there. |-joined formats + pure parsers.
  2. {daemon_url} rendered from the requested address — a port-0 daemon sent every pane's hooks to http://127.0.0.1:0. Resolved before the router is built.
  3. A resumed or prompt-less harness sat in starting foreverSessionStart flipped the source to hooks (stopping the scraper) but mapped to nothing. SessionStart on a starting row ⇒ idle.
  4. A held row flapped — the supervisor saw its dead tmux session as a crash and scheduled a resume the guard then refused, every backoff. Held rows are left alone.

CI

pr-checks.yml: tmux on the Linux leg and SMOOTH_E2E_STRICT=1 so a missing tmux/bash/curl/th fails the job instead of passing it having run nothing. Windows compiles the suite empty (#![cfg(unix)]). The suite runs inside the normal cargo nextest run --profile ci.

Verification

  • cargo nextest run -p smooai-smooth-daemon --test flow_e2e: 24/24, 68.5 s wall on a dev box at load 25–40
  • cargo test -p smooai-smooth-flow --lib: 75/75 (new: pane parsers, SessionStart ⇒ idle, held rows survive a tick, prefs test made PATH-independent)
  • cargo test -p smooai-smooth-daemon --lib operator flow_route: green (new: ephemeral port resolution)
  • cargo fmt --check, cargo clippy (no new warnings), pnpm format:check

Docs

docs/Engineering/SmoothFlow-Testing.md — the cmux/orca survey, the layers table, a per-test table, the rig, the fake-agent contract, runtimes, the CI split rule, gaps. Linked from docs/Architecture/SmoothFlow.md § Testing.

Pearl: th-8e3087 (epic th-6ac036). Sibling: th-a58a97 (#548, the macOS XCUITest lane).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q5YUmVQ5obDT27oLjgt7AL

… hooks contract, th flow

Brent asked for cmux/orca-style tests. Lane A had engine unit tests, the app
its XCTests, the phones codec tests — nothing booted the real thing and
drove it like a client. This adds crates/smooth-daemon/tests/flow_e2e: a REAL
smooth-daemon per test (isolated HOME, ephemeral port, private tmux server,
no lock / tailscale / relay / gateway creds), driven over the flow WS, the
HTTP siblings, the `th` binary and POST /api/flow/hooks. The agent under
test is tests/fixtures/fake-agent, installed through harness manifests the
way any coding CLI is, in four flavours — hooks / learned id / native /
scrape — one per state source the engine supports. 24 tests, ~70 s wall.

The suite found four engine bugs on its first run, fixed here:

- tmux rewrites control characters under a non-UTF-8 locale, so the
  tab-joined `#{pane_dead}\t#{pane_dead_status}` query read `1_2` and a dead
  pane was NEVER detected for any daemon not started from a shell (launchd,
  CI). Every supervision rule was inert there. `|`-joined + pure parsers.
- `{daemon_url}` was rendered from the requested address, so a port-0 daemon
  sent every pane's hooks to http://127.0.0.1:0. Resolved before the router
  is built.
- A resumed or prompt-less harness sat in `starting` forever: SessionStart
  flipped the source to `hooks` (stopping the scraper) and mapped to
  nothing. SessionStart on a starting row is now idle.
- A `held` row (duplicate-resume guard) was flapped by the supervisor, which
  saw its dead tmux session as a crash and scheduled a resume the guard then
  refused, every backoff. Held rows are left alone.

CI: tmux on the Linux leg + SMOOTH_E2E_STRICT=1 so a skip is a failure;
Windows compiles the suite empty. docs/Engineering/SmoothFlow-Testing.md is
the map (survey, layers, per-test table, the rig, fake-agent, runtimes, gaps).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5YUmVQ5obDT27oLjgt7AL
@brentrager
brentrager enabled auto-merge (squash) September 9, 2026 14:32
@changeset-bot

changeset-bot Bot commented Sep 9, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 22939e2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@smooai/smooth Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

brentrager and others added 6 commits September 9, 2026 10:40
… hooks contract, th flow

Brent asked for cmux/orca-style tests. Lane A had engine unit tests, the app
its XCTests, the phones codec tests — nothing booted the real thing and
drove it like a client. This adds crates/smooth-daemon/tests/flow_e2e: a REAL
smooth-daemon per test (isolated HOME, ephemeral port, private tmux server,
no lock / tailscale / relay / gateway creds), driven over the flow WS, the
HTTP siblings, the `th` binary and POST /api/flow/hooks. The agent under
test is tests/fixtures/fake-agent, installed through harness manifests the
way any coding CLI is, in four flavours — hooks / learned id / native /
scrape — one per state source the engine supports. 24 tests, ~70 s wall.

The suite found four engine bugs on its first run, fixed here:

- tmux rewrites control characters under a non-UTF-8 locale, so the
  tab-joined `#{pane_dead}\t#{pane_dead_status}` query read `1_2` and a dead
  pane was NEVER detected for any daemon not started from a shell (launchd,
  CI). Every supervision rule was inert there. `|`-joined + pure parsers.
- `{daemon_url}` was rendered from the requested address, so a port-0 daemon
  sent every pane's hooks to http://127.0.0.1:0. Resolved before the router
  is built.
- A resumed or prompt-less harness sat in `starting` forever: SessionStart
  flipped the source to `hooks` (stopping the scraper) and mapped to
  nothing. SessionStart on a starting row is now idle.
- A `held` row (duplicate-resume guard) was flapped by the supervisor, which
  saw its dead tmux session as a crash and scheduled a resume the guard then
  refused, every backoff. Held rows are left alone.

CI: tmux on the Linux leg + SMOOTH_E2E_STRICT=1 so a skip is a failure;
Windows compiles the suite empty. docs/Engineering/SmoothFlow-Testing.md is
the map (survey, layers, per-test table, the rig, fake-agent, runtimes, gaps).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5YUmVQ5obDT27oLjgt7AL
… a state transition

A real CLI without credentials sits on onboarding in the fresh HOME — a
pane no scrape pattern matches, so `starting` is the engine's honest
reading; th code reports natively only once it can boot.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5YUmVQ5obDT27oLjgt7AL
…ness-flake fix

Also lands the rig's own daemon.addr (a second daemon deliberately does not
advertise itself since #546) and the th-code providers.json opt-in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…'s hold sticks

The duplicate-resume guard was racy end to end, not just in the tick: kill,
supervise_one and POST /api/flow/hooks each read a session row and then wrote
it, so a `held` written between another reader's read and its write was
silently overwritten (~1 in 3 under load). The observed sequence: kill --resume
holds row A, A's dying agent posts its last Stop, the hook — resolved before
the hold landed — writes A back to idle, and the next supervision pass reads
A's dead tmux session as an unexpected crash.

All three paths now take the same per-session lock (supervision try_locks and
skips: a kill owns the row) and re-read the row under it, holding drops any
pending relaunch backoff, and a hook naming a held row is ignored — its harness
session id belongs to the live process that holds it, not to this row.

15/15 green on the previously 1-in-3 flaky e2e test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The remote tip was an older squash of this same work on an older base. This
worktree carries it plus the main merge and the rule-4 race fix, so the tree
is kept as-is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-session lock already fixes this path, but nothing pinned it: the bug
was pure ordering — hook read the row BEFORE relaunch wrote Starting, saw the
pre-kill idle, and HookOutcome::Started's `state == Starting` test declined, so
a resumed row sat at `starting` until something timed out. The new test
simulates the kill's slow half by holding the session lock rather than sleeping
out KILL_GRACE, so it pins the ordering, not the timing; it fails on the
pre-lock read and passes now.

Narrowing kill's critical section so a hook can never be delayed past its own
client timeout is deferred to th-786fdc: the attempt destabilised
shell_lifecycle (8 failures in 16 runs against 0 in 16), with the shell logging
out at the second-client attach — before kill is reached at all — which points
at a separate race in the PTY detach/attach bridge.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant