diff --git a/example/src/Examples/SurfaceExample.tsx b/example/src/Examples/SurfaceExample.tsx index e941c7c843..4a745e341e 100644 --- a/example/src/Examples/SurfaceExample.tsx +++ b/example/src/Examples/SurfaceExample.tsx @@ -53,7 +53,7 @@ const SurfaceExample = () => { return ( - + { - + -realistic-1-t0.02.png`). Human visibility: borderline — noticeable only when flipping between the two images. + +## Caveats + +- **The crop is not perfectly isolated.** Under the gross break a full-width 33 px band at the top of the crop changed: the Appbar is itself a `Surface`, and the break changed every Surface. The realistic break showed no bleed. An Appbar-only change could register against this crop; that is inherent in screenshotting real screens rather than isolated components. +- **Dev-client chrome can land in the crop.** The Expo dev-client's floating "Tools" button, switched on by a stray dev-menu press, sat inside the Android section and produced a deterministic 2,822 px false FAIL with `src/` clean (`evidence/diff-images/android-devclient-tools-button.png`, `evidence/devclient-excerpt.json`). Turning it off in the dev menu ("Tools button") restored 0. A release build would remove the whole class of dev menu, dev launcher and floating button. +- **Not measured:** swiftshader (what `ubuntu-latest` renders with), cold simulator/emulator boot, another host or day, runtime or image updates, other components, text-heavy crops, dark theme, web (`--crop-on` is refused there). + +## Reproducing by hand + +Prerequisites: the example app built and installed on the device pinned in `env.json`, Metro running (`yarn example start`). Commands verified on 2026-09-10/11. agent-device sessions are keyed by cwd and bound to one device — use one directory per platform, or `--session ` for the second. + +iOS (from the same cwd every time): + +```bash +UDID=2464A356-C17C-4B0D-99DB-CDFBDB98826C +npx agent-device@0.21.0 open com.callstack.reactnativepaperexample --relaunch --platform ios --udid $UDID --json +# the dev menu appears a few seconds after relaunch; wait for it, then: +npx agent-device@0.21.0 wait stable 500 10000 --platform ios --udid $UDID --json +npx agent-device@0.21.0 press 'label="Close"' --platform ios --udid $UDID --json +# navigate to Surface (the app restores its last screen; if it is not Surface, press Back to the list and press the "Surface" row) +npx agent-device@0.21.0 wait stable 500 10000 --platform ios --udid $UDID --json && sleep 2 +npx agent-device@0.21.0 screenshot current.png --crop-on 'id="surface-example-elevated"' --pixel-density 3 --platform ios --udid $UDID --json +npx agent-device@0.21.0 diff screenshot --baseline example/visual/__baselines__/ios/surface-example-elevated.png current.png --out diff.png --threshold 0.02 --json +``` + +Android (`--session android --platform android` on every command; after `open --relaunch` the dev launcher shows — press the row labelled with the Metro URL, e.g. `http://192.168.1.151:8081`, then wait for the app to load; no `--pixel-density`): + +```bash +npx agent-device@0.21.0 open com.callstack.reactnativepaperexample --relaunch --platform android --session android --json +npx agent-device@0.21.0 screenshot current.png --crop-on 'id="surface-example-elevated"' --platform android --session android --json +npx agent-device@0.21.0 diff screenshot --baseline example/visual/__baselines__/android/surface-example-elevated.png current.png --out diff.png --threshold 0.02 --json +``` + +To see a failure: in `src/components/Surface.tsx` change `shadow(elevation, …)` to `shadow(elevation === 1 ? 2 : elevation, …)` (iOS) or `androidElevationLevels[elevation]` to `androidElevationLevels[elevation === 1 ? 2 : elevation]` (Android), relaunch the app so it fetches the bundle, capture, diff — expect ~10,179 px (iOS) / ~9,336 px (Android) at 0.02 and `match: true` at 0.1. `git checkout -- src/components/Surface.tsx` afterwards. + +## What a real runner would have to handle + +Learned the hard way while trying to script the loop above; each is documented in `evidence/issues.md` and implemented on the runner branch, none of it is in this PR: + +- **Relaunch before every capture.** Fast Refresh silently stopped reaching the Android app; without a fresh bundle a stale screen reads as PASS. +- **The example app persists navigation state.** A relaunch lands on the last screen, so "start at the top of the list" is false; the runner has to press Back to the root and pick the Surface _row_, not the header title. +- **Dev-client chrome:** the dev menu appears seconds _after_ the app is ready, the Android relaunch lands in the dev launcher, and the floating Tools button can appear inside the crop. +- `snapshot` returns an empty `nodes` array when nothing changed unless `--force-full` is passed. +- Set `--threshold` explicitly (0.02 for soft shadows) and verify noise at it; the default hides real regressions. +- Pin the device (UDID, runtime, density) and refuse to compare or re-baseline on anything else; derive expected capture size from the baseline PNG, per story. +- Plain `.mjs` entry guarded by `import.meta.main` does nothing on Node 20/22; guard with an `argv[1]` comparison. + +## agent-device issues found + +Nineteen, with commands and evidence, in `evidence/issues.md`. The ones that matter most: the default diff threshold misses soft-shadow regressions (4); `find` taps what it finds (1); sessions are bound to one device per cwd (6); a dev-build screenshot can contain dev-client chrome the diff cannot tell apart from the app (19). + +## Evidence + +Every number above resolves to a row or a file: + +- `evidence/results.csv` — one row per `diff screenshot` run (72 rows): platform, capture, threshold, total and changed pixels, mismatch %, regions, match, and the name of the raw JSON it came from. The raw per-command JSON files are on the runner branch. +- `evidence/diff-images/` — one diff image per platform for the realistic break (ring on the Elevation 1 card) and the gross break, both at 0.02, plus the dev-client Tools-button false FAIL. No 0.1 images exist because `diff screenshot --out` writes nothing on a match (issue 18). +- `evidence/a11y-excerpt.json`, `evidence/devclient-excerpt.json` — the nodes that matter from the accessibility trees; full trees on the runner branch. +- `env.json` — the pinned devices, versions and thresholds. + +The first-run rows predate the test-id rename and their source files say `surface-elevated` / `surface-flat`; renaming an id changes no pixel. diff --git a/example/visual/__baselines__/android/surface-example-elevated.png b/example/visual/__baselines__/android/surface-example-elevated.png new file mode 100644 index 0000000000..c94e2bc1b1 Binary files /dev/null and b/example/visual/__baselines__/android/surface-example-elevated.png differ diff --git a/example/visual/__baselines__/android/surface-example-flat.png b/example/visual/__baselines__/android/surface-example-flat.png new file mode 100644 index 0000000000..4f688b4e61 Binary files /dev/null and b/example/visual/__baselines__/android/surface-example-flat.png differ diff --git a/example/visual/__baselines__/ios/surface-example-elevated.png b/example/visual/__baselines__/ios/surface-example-elevated.png new file mode 100644 index 0000000000..5eb2aa499f Binary files /dev/null and b/example/visual/__baselines__/ios/surface-example-elevated.png differ diff --git a/example/visual/__baselines__/ios/surface-example-flat.png b/example/visual/__baselines__/ios/surface-example-flat.png new file mode 100644 index 0000000000..b909a990ea Binary files /dev/null and b/example/visual/__baselines__/ios/surface-example-flat.png differ diff --git a/example/visual/env.json b/example/visual/env.json new file mode 100644 index 0000000000..453b33ef14 --- /dev/null +++ b/example/visual/env.json @@ -0,0 +1,47 @@ +{ + "platform": "ios", + "device": "iPhone 17 Pro", + "udid": "2464A356-C17C-4B0D-99DB-CDFBDB98826C", + "iosVersion": "26.5", + "runtime": "com.apple.CoreSimulator.SimRuntime.iOS-26-5", + "scale": 3, + "pixelDensity": 3, + "agentDeviceVersion": "0.21.0", + "reactNativeVersion": "0.85.3", + "branch": "poc/agent-device-visual", + "date": "2026-09-10T09:30:58Z", + "baselines": { + "surface-example-elevated": "__baselines__/ios/surface-example-elevated.png", + "surface-example-flat": "__baselines__/ios/surface-example-flat.png" + }, + "diffThresholds": { + "cliDefault": 0.1, + "usedForShadowRegressions": 0.02 + }, + "android": { + "avd": "Pixel_10_Pro", + "device": "Pixel 10 Pro", + "serial": "emulator-5554", + "apiLevel": 37, + "androidRelease": "17", + "systemImage": "system-images/android-37.0/google_apis_playstore_ps16k/arm64-v8a/", + "density": 480, + "screenPx": "1280x2856", + "emulatorVersion": "36.6.11.0 (build_id 15507667)", + "gpuMode": "auto (hw.gpu.enabled=yes, hw.gpu.mode=auto; swiftshader NOT measured)", + "agentDeviceVersion": "0.21.0", + "agentDeviceSession": "android", + "reactNativeVersion": "0.85.3", + "metroHost": "http://192.168.1.151:8081", + "date": "2026-09-10T09:45:00Z", + "baselines": { + "surface-example-elevated": "__baselines__/android/surface-example-elevated.png", + "surface-example-flat": "__baselines__/android/surface-example-flat.png", + "pixelDensityFlag": "unsupported on Android (screenshots are native device pixels)" + }, + "diffThresholds": { + "cliDefault": 0.1, + "usedForShadowRegressions": 0.02 + } + } +} diff --git a/example/visual/evidence/a11y-excerpt.json b/example/visual/evidence/a11y-excerpt.json new file mode 100644 index 0000000000..d2d253df10 --- /dev/null +++ b/example/visual/evidence/a11y-excerpt.json @@ -0,0 +1,74 @@ +{ + "about": "Excerpts from the XCUITest / UiAutomator trees captured 2026-09-10 with `npx agent-device@0.21.0 snapshot --raw --force-full --json`. The full trees (13,500 lines) are on branch poc/agent-device-visual-runner under example/visual/evidence/a11y/. Test id at the time: surface-elevated (renamed surface-example-elevated later; the rename changes no pixel).", + "ios": { + "with_accessible": { + "total_nodes": 115, + "elevation_labelled_nodes": 2, + "surface_section_node": [ + { + "type": "Other", + "role": "RCTViewComponentView", + "identifier": "surface-elevated", + "label": "Elevated surface, Elevation 0, Elevation 1, Elevation 2, Elevation 3, Elevation 4, Elevation 5, Vertical scroll bar, 3 pages", + "rect": { + "x": 0, + "y": 134, + "width": 402, + "height": 214 + } + } + ] + }, + "testID_only": { + "total_nodes": 177, + "elevation_labelled_nodes": 26, + "surface_section_node": [ + { + "type": "Other", + "role": "RCTViewComponentView", + "identifier": "surface-elevated", + "rect": { + "x": 0, + "y": 134, + "width": 402, + "height": 214 + } + } + ] + } + }, + "android": { + "with_accessible": { + "total_nodes": 217, + "elevation_labelled_nodes": 13, + "surface_section_node": [ + { + "type": "android.view.ViewGroup", + "identifier": "surface-elevated", + "rect": { + "x": 0, + "y": 372, + "width": 1280, + "height": 642 + } + } + ] + }, + "testID_only": { + "total_nodes": 217, + "elevation_labelled_nodes": 13, + "surface_section_node": [ + { + "type": "android.view.ViewGroup", + "identifier": "surface-elevated", + "rect": { + "x": 0, + "y": 372, + "width": 1280, + "height": 642 + } + } + ] + } + } +} \ No newline at end of file diff --git a/example/visual/evidence/devclient-excerpt.json b/example/visual/evidence/devclient-excerpt.json new file mode 100644 index 0000000000..33a0499075 --- /dev/null +++ b/example/visual/evidence/devclient-excerpt.json @@ -0,0 +1,42 @@ +{ + "about": "Excerpts captured 2026-09-11 with `snapshot --force-full --json`; full files on branch poc/agent-device-visual-runner under example/visual/evidence/runner/.", + "ios_after_open_relaunch": { + "immediately": { + "total_nodes": 3, + "labels": [ + "React Native Paper Example", + null, + "Downloading 100%\u2026" + ] + }, + "six_seconds_later": { + "total_nodes": 77, + "dev_menu_close_node": [ + { + "type": "Button", + "role": "SwiftUI.AccessibilityNode", + "identifier": "xmark", + "label": "Close", + "rect": { + "x": 344.06965174129357, + "y": 401.5837479270315, + "width": 34.56716417910445, + "height": 34.56716417910445 + } + } + ] + } + }, + "android_devclient_floating_tools_button": [ + { + "type": "android.widget.ImageView", + "label": "Tools", + "rect": { + "x": 1115, + "y": 243, + "width": 78, + "height": 78 + } + } + ] +} \ No newline at end of file diff --git a/example/visual/evidence/diff-images/android-devclient-tools-button.png b/example/visual/evidence/diff-images/android-devclient-tools-button.png new file mode 100644 index 0000000000..856b41c00b Binary files /dev/null and b/example/visual/evidence/diff-images/android-devclient-tools-button.png differ diff --git a/example/visual/evidence/diff-images/android-gross-t0.02.png b/example/visual/evidence/diff-images/android-gross-t0.02.png new file mode 100644 index 0000000000..a6f157ad1a Binary files /dev/null and b/example/visual/evidence/diff-images/android-gross-t0.02.png differ diff --git a/example/visual/evidence/diff-images/android-realistic-1-t0.02.png b/example/visual/evidence/diff-images/android-realistic-1-t0.02.png new file mode 100644 index 0000000000..5e6290efe8 Binary files /dev/null and b/example/visual/evidence/diff-images/android-realistic-1-t0.02.png differ diff --git a/example/visual/evidence/diff-images/ios-gross-t0.02.png b/example/visual/evidence/diff-images/ios-gross-t0.02.png new file mode 100644 index 0000000000..79e8aa8658 Binary files /dev/null and b/example/visual/evidence/diff-images/ios-gross-t0.02.png differ diff --git a/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png b/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png new file mode 100644 index 0000000000..02219e89f4 Binary files /dev/null and b/example/visual/evidence/diff-images/ios-realistic-1-t0.02.png differ diff --git a/example/visual/evidence/issues.md b/example/visual/evidence/issues.md new file mode 100644 index 0000000000..667f1db1fd --- /dev/null +++ b/example/visual/evidence/issues.md @@ -0,0 +1,360 @@ +# agent-device dogfooding issues + +Tool: `agent-device` 0.21.0 (run as `npx agent-device@0.21.0`), Node v24.18. +Device: iPhone 17 Pro, iOS 26.5, UDID `2464A356-C17C-4B0D-99DB-CDFBDB98826C`. +Session state dir (for runner/request logs): `/Users/juliankobrynski/.agent-device/sessions/cwd_4a6658db6e09aaf6_default/`. + +--- + +## 1. `find` taps the match instead of just locating it + +`find` is documented/named as a query, but a successful match performs a tap, so +navigation happens as a side effect. There is no way to locate an element without +acting on it (you have to fall back to `snapshot`). + +Command: + +``` +npx agent-device@0.21.0 find 'label="Surface"' --udid 2464A356-C17C-4B0D-99DB-CDFBDB98826C --json +``` + +Output (note `"Tapped …"`): + +```json +{ + "success": true, + "data": { + "ref": "@e13", + "locator": "any", + "query": "label=\"Surface\"", + "x": 201, + "y": 501, + "message": "Tapped @e13 (201, 501)" + } +} +``` + +Artifact: `19-find-surface.json` (also `16-find.json`, `17-find.json`). + +Related: when the label is not on screen, `find` fails with a generic +`find did not match any element` and no indication that the element may simply be +below the fold — artifact `17-find.json`. + +--- + +## 2. `scroll bottom` hits a safety limit and leaves the Expo dev menu open + +`scroll bottom` on the example-list screen never reports reaching the edge, aborts +on a safety limit, and the repeated scrolling gesture is interpreted by the Expo +dev client as the "open dev menu" gesture. The app is then left with the dev-menu +sheet on top, which silently poisons any screenshot taken afterwards. + +Command: + +``` +npx agent-device@0.21.0 scroll bottom --udid 2464A356-C17C-4B0D-99DB-CDFBDB98826C --json +``` + +Output: + +```json +{ + "success": false, + "error": { + "code": "COMMAND_FAILED", + "message": "scroll bottom reached the safety limit before the snapshot showed the edge", + "hint": "The scoped scroll container still reports hidden content. Use a smaller manual scroll + snapshot loop to inspect the current state." + } +} +``` + +Artifact: `16-scroll.json`. Recovery: `press 'label="Close"'` (artifact `18-press-close.json`). +Workaround used for the PoC: `scroll down 6 --settle` twice instead of `scroll bottom`. + +--- + +## 3. `open --relaunch` leaves the Expo dev menu open on the restarted app + +New in Task B. After `open … --relaunch` the app process is genuinely restarted +(PID 27223 -> 37870) and React Navigation restores the previous screen, but the +Expo dev-client menu sheet is on screen. A capture taken right after the relaunch +would include it. There is no signal in the `open --json` payload that the app is +not in a clean state; only a `snapshot` reveals it. + +Command: + +``` +npx agent-device@0.21.0 open com.callstack.reactnativepaperexample --platform ios \ + --udid 2464A356-C17C-4B0D-99DB-CDFBDB98826C --relaunch --json +``` + +`open` reports success (`startup.durationMs: 1032`), artifact `45-open-relaunch.json`. +The following snapshot shows the dev menu nodes (`Runtime version: exposdk:56.0.0`, +`Close`, `Reload`, `Go home`) — artifact `47-snapshot.json`. +Recovery: `press 'label="Close"'` — artifact `48-press-close.json`. + +--- + +## 4. Default `diff screenshot --threshold 0.1` misses real soft-shadow regressions + +New in Task B, and the most consequential finding for using this tool as a visual +regression guard for Material elevation. + +A realistic regression — Surface elevation level 1 rendering the level 2 spot and +ambient shadow, one step off — is reported as a **perfect match** at the default +threshold: + +``` +npx agent-device@0.21.0 diff screenshot \ + --baseline example/visual/__baselines__/ios/surface-example-elevated.png \ + --out --threshold 0.1 --json +``` + +Committed evidence: `evidence/results.csv` rows `ios / realistic / 0.1` (0 changed pixels, +`match: true`) and `ios / realistic / 0.02` (10,179 changed pixels) are the two diffs of the +same capture; the three repeats are the `ios / realistic-{1,2,3}` rows at both thresholds; +the diff image at 0.02 is `evidence/diff-images/ios-realistic-1-t0.02.png`. +There is no 0.1 image because `diff screenshot --out` writes nothing on a match (issue 18). +The raw capture PNG was not committed. + +```json +{ + "success": true, + "data": { + "totalPixels": 774252, + "differentPixels": 0, + "mismatchPercentage": 0, + "match": true + } +} +``` + +The same pair at `--threshold 0.02` reports 10179 different pixels (1.31%) in a +single 378x378 region exactly around the affected surface (`results.csv`, +`ios / realistic`; image `ios-realistic-1-t0.02.png`). + +The gross break (every elevated surface forced to the level 5 shadow) is also +badly under-reported at the default threshold: 4277 px / 0.55% in one 316x17 band +versus 116292 px / 15.02% in 3 regions at 0.02 (`results.csv`, `ios / gross`; +image `ios-gross-t0.02.png`). At the default threshold, a change that is +plainly visible to the eye across the whole screen registers as a thin sliver. + +Suggestion: either lower the default, or document that the default threshold is +unsuitable for low-contrast/anti-aliased content such as shadows and gradients, +and surface the threshold actually used in the JSON output (it is not echoed today). + +--- + +## 5. Minor: no way to time a capture from the JSON output + +`screenshot --json` returns path and dimensions but no timing, so wall-clock has to +be measured around the process. Measured externally for the record: single +`screenshot --crop-on … --pixel-density 3` = 673-737 ms; `open --relaunch` = 1539 ms +round-trip (`startup.durationMs` 1032). Artifacts `44-screenshot.json`, +`45-open-relaunch.json`. + +--- + +# Android (Fri 2026-09-10, agent-device 0.21.0, Pixel_10_Pro API 37) + +## 6. A cwd whose session is bound to one platform cannot target another device + +`boot --platform android --device Pixel_10_Pro` from a cwd whose default session was +bound to an iOS simulator fails with `INVALID_ARGS` ("already bound to apple device +… but this request selected --platform=android"). Artifact `74-boot.json`. The hint is +good (pass a different `--session`), and `--session android` worked (`75-boot.json`), +but a session keyed only by cwd means one checkout cannot drive iOS and Android +without inventing session names by hand. Worth documenting in the multi-platform +guide, or keying sessions by cwd+platform. + +## 7. The same `find` selector is not portable across platforms + +`find 'label="Surface"'` resolves to exactly one node on iOS, but on Android it fails +with `AMBIGUOUS_MATCH` — the accessibility tree exposes both the row `ViewGroup` and +its child `TextView` with the same label (`83-find-surface.json`, +candidates `@e21 [group] "Surface"`, `@e22 [text] "Surface"`). Any cross-platform +script has to carry two selectors for one row. A role-defaulting rule (prefer the +tappable ancestor) or a documented `role=` qualifier for the common case would fix it. + +## 8. Refs from an earlier snapshot are rejected with a confusing message + +`press '@e21'`, with the ref taken from the immediately preceding `snapshot --json`, +failed: "Ref @e21 needs a complete snapshot — the current frame only authorizes its +emitted refs" (`84-press-surface.json`). Re-snapshotting and using the new ref for the +same element (`@e55`) worked (`86-press-surface.json`). The message does not say what +invalidated the ref or that a fresh snapshot renumbers every element. + +## 9. Android relaunch lands in the Expo dev launcher, not the app + +Android version of issue 3. `open com.callstack.reactnativepaperexample --platform +android` on an app whose process is alive shows the Expo **dev menu** overlay — two +presses ("Continue", then "Close") were needed to reach the app (`77-`/`79-`/ +`81-snapshot.json`). After `open --relaunch` (pid 9944 → 10860, `95-open-relaunch.json`) +the app starts on the Expo dev launcher screen and must be reconnected by pressing the +"RECENTLY OPENED" entry (`97-press-recent.json`), then the JS bundle takes several +seconds to load — `wait stable` returns immediately on the "nearly-empty tree" and only +`wait text` (or a retry) reveals the app is still on "Connecting to the development +server…" (`98-wait.json`). A dev-client aware `open` (or a documented recipe) would +remove three commands from every relaunch. + +## 10. `logs` reports success on an empty log unless `logs start` ran first + +`logs --json` returned `success: true` with `sizeBytes: 0`, `active: false`, +`state: "inactive"` (`104-logs.json`) — no indication that nothing was being captured. +Confirming the Fast Refresh probe had to be done with `adb logcat -d | grep` +(19 hits). `logs start` (`105-logs-start.json`) arms capture, but a success response +for a log that is not running is misleading. + +## 11. Recorded, not a bug: `--pixel-density` is iOS-only + +`screenshot --crop-on … --pixel-density 3` on Android → `UNSUPPORTED_OPERATION`, +"currently supported only on iOS-family simulators" (`91-screenshot-pd3.json`). +Android already returns native device pixels (1280x642 for the crop at 480 dpi), so +nothing is lost — but the iOS default of 1x logical points remains an easy trap. + +## Threshold data point for issue 4 (Android) + +Android repeats the iOS finding for the realistic break: 0 changed pixels and +`match: true` at the default 0.1, 9,336 px at 0.02 (`results.csv`, `android / realistic`). +Unlike iOS, the gross break is caught at the default (65,051 px / 7.92 %). + +## 12. The typed Node client is exported but unreachable without a dependency + +`agent-device@0.21.0`'s `package.json` does export the client +(`exports["."] → dist/src/index.js`, which exports `createAgentDeviceClient`), so +the plan's "one Node file on `createAgentDeviceClient()`" is supported in +principle. In practice the package only exists in the `npx` cache +(`~/.npm/_npx//node_modules/agent-device`); `import('agent-device')` from +`example/` fails with `ERR_MODULE_NOT_FOUND`. Reaching the typed API therefore +means adding a devDependency, which the PoC forbade, so the runner (on branch `poc/agent-device-visual-runner`) +spawns the CLI with `--json` instead. Worth a documented "run the client without +installing" story (or a `npx agent-device init`-style scaffold), since the CLI +path costs an `npx` resolution per command and loses all the result types. + +## 13. `snapshot` returns an empty `nodes` array when the tree is unchanged + +A plain `snapshot` only reports nodes that changed since the previous snapshot in +the same session. When nothing changed it returns `data.nodes: []` with +`success: true`, which is indistinguishable from "the screen is empty" or "the +node is gone" for any caller that greps the result for an identifier. This cost +real time in the review round: a plain snapshot after the `testID` edit looked +as if the id had never reached the tree, when it had been there the whole time — +`--force-full` returned it (the resulting trees are excerpted in +`evidence/a11y-excerpt.json`; full files on branch `poc/agent-device-visual-runner`). Anything that asks "is this node +on screen?" must pass `--force-full`; the runner on that branch does so for +exactly this reason. +An empty diff-mode result would be much less of a trap if the payload said so — +e.g. a `mode: "diff"` / `unchanged: true` field alongside the empty array. + +## 14. `open --relaunch` restores the app's previous route + +The first scripted loop assumed a relaunch puts the example list back at the +top, so it could scroll down a fixed number of rows to reach "Surface". It does +not. The example app persists its navigation state in AsyncStorage +(`PERSISTENCE_KEY = 'NAVIGATION_STATE'` in `example/src/index.tsx`), so after +`open --relaunch` (iOS) or `am force-stop` + `open` (Android) the app comes back +on whatever screen it was on. Reproduced on both platforms in this pass: + +The logs of the runs that first showed this were not kept. The fixed runner on +branch `poc/agent-device-visual-runner` logs `relaunching the app … not on the Surface screen — going back to the +example list … pressing "Back" … at the example list root` when started from +another example screen, on both platforms — a branch that could not be reached if a +relaunch reset the route. Its run summaries are committed on that branch under +`example/visual/evidence/runs/`. + +This is a runner bug, not an agent-device bug: navigation has to be driven +back to the list explicitly (press `Back` until no `Back` control is left, then +scroll for the row), not implied by a relaunch. Related trap: because +`navigateToSurface()` matches on `label === 'Surface'` and picks the largest +match, on the restored _Surface_ screen it pressed the nav header title +(`RCTParagraphComponentView "Surface"`) instead of a list row. That log was not +kept either; the trap is encoded as a unit test in the runner on branch `poc/agent-device-visual-runner` +(`findSurfaceRow picks the list row, not the Appbar title`), whose fix picks the +row by position and width. + +## 15. The dev-menu check runs before the app finishes loading + +`dismissDevMenu()` snapshots once, immediately after the relaunch. On iOS that +snapshot came back with three nodes — `UIApplication`, `SplashScreenLogo`, +`Downloading 100%…` — so it found none of the `Close` / `Continue` labels and +returned "clean". Reproduced for the record on 2026-09-11 — +`evidence/devclient-excerpt.json`, `ios_after_open_relaunch`: immediately after +`open --relaunch` the tree is 3 nodes (`React Native Paper Example`, the splash +image, `Downloading 100%…`); six seconds later it is 77 nodes including the dev +menu's `Close` (`identifier: "xmark"`). Full snapshots on branch `poc/agent-device-visual-runner` under +`example/visual/evidence/runner/`. The dev menu appears _after_ the bundle finishes downloading; a +check that snapshots once at open time cannot see it, and it then swallows every +subsequent scroll. The check has to +wait for the app to be ready first, and be repeated inside the navigation loop +rather than run once. + +On Android the same step fails differently and confirms issue 9: after +`am force-stop` + `open`, the app is the Expo **dev launcher** +(`DEVELOPMENT SERVERS` / `Fetch development servers` / `RECENTLY OPENED`), whose +labels differ from the dev menu's. Recovery is to press the recently-opened +entry (`http://192.168.1.151:8081`), which reloads the bundle in ~20 s. + +## 16. Fast Refresh did not reach the Android app; a manual Reload was required + +On iOS every edit to `src/components/Surface.tsx` was picked up within a few +seconds (first realistic-break capture already differed: 10,179 px). On Android +the same edits produced **zero** changed pixels — including the gross break, +every elevation level forced to 5 — even though Metro was serving the new code +(`curl .../.expo/.virtual-metro-entry.bundle?platform=android` contained the +probe added alongside the break) and `adb reverse --list` showed +`tcp:8081 tcp:8081`. `adb logcat` carried no React Native output at all, so it +is useless as a "did the edit land" probe here. Pressing **Reload** in the dev +menu (`adb shell input keyevent 82`, then press the `Reload` node) made the +break appear immediately at the expected magnitude (gross: 65,051 px at 0.1). +Consequence for the loop: on Android, an edit must be followed by an explicit +reload — a device-runner that relies on Fast Refresh will silently compare stale +pixels and report PASS. + +## 17. Minor: an agent-device command occasionally exits non-zero with no output + +Twice in this pass (`press '@e2'`, `press 'label="Close"'`, both on iOS) the CLI +exited 1 with completely empty stdout _and_ stderr. Re-running the identical +command immediately afterwards succeeded. Nothing to act on beyond noting that a +caller cannot distinguish this from a crash, and that a runner's "parse the +first `{`..`}` out of stdout" fallback turns it into a `null` response rather +than a clear error. + +## 18. Minor: `diff screenshot --out` writes nothing when the images match + +`--out` is silently ignored on a `match: true` diff, and the response omits +`diffPath` as well. That is defensible (there is nothing to draw), but it means +"the diff image exists" cannot be used as evidence that a diff ran, and a +per-threshold sweep produces a gappy set of files: the realistic break has a +diff image at 0.02 but none at 0.1, because at 0.1 it matches. The 0.1 rows in +`evidence/results.csv` are the record for those. + +## 19. The dev-client's floating "Tools" button lands inside the crop and reads as a regression + +Found 2026-09-11 while re-running the Android loop with `src/` clean after the +round-2 changes. The scripted loop reported a deterministic **FAIL** — +`surface-example-elevated changed=2822 (0.34%) regions=1` — twice in a row, +against a baseline byte-identical to the committed one (`md5 8693c919…` in both +the working tree and `HEAD`). The diff image +(`evidence/diff-images/android-devclient-tools-button.png`) shows the change: a +152x74 pill at the top-right of the crop reading **Tools** — the Expo dev-client's +floating dev-tools button, `android.widget.ImageView` label `Tools`, rect +`{x:1115, y:243, w:78, h:78}` (`evidence/devclient-excerpt.json`, +`android_devclient_floating_tools_button`). Both failing run summaries are on branch `poc/agent-device-visual-runner`. + +It is a dev-client setting, toggled by the dev menu entry **"Tools button"** +(the dev menu lists `Close`, `Reload`, `TOOLS`, `Tools button`, `Toggle Dev Menu`, …). +Pressing it and then `Close` removed the node (60 nodes, no `Tools`), and the next +run passed with `changed=0` on both stories. The setting +survives a relaunch. It had been off in every earlier run today +(`round2-pass`, 08:53 UTC, 0 px) and was on by 10:14 UTC; which earlier dev-menu +interaction flipped it is not known. + +Two lessons. For agent-device: a screenshot of a dev build can contain dev-client +chrome that is not part of the app, and `diff screenshot` cannot tell — worth a +note in the docs, or a `screenshot` option that hides dev-client overlays. For +any runner: a dev overlay inside the crop must never be reported as a visual +regression; the runner on branch `poc/agent-device-visual-runner` looks for the floating node after dismissing the +dev menu and launcher, turns it off through the dev menu when it can, and +otherwise stops with a message naming the node and the manual fix. This is also +the strongest argument so far for capturing from a release build rather than a +dev-client once this moves past a PoC. diff --git a/example/visual/evidence/results.csv b/example/visual/evidence/results.csv new file mode 100644 index 0000000000..5e8efe206c --- /dev/null +++ b/example/visual/evidence/results.csv @@ -0,0 +1,73 @@ +platform,group,capture,threshold,totalPixels,differentPixels,mismatchPercent,regions,match,source +ios,first-run,a-vs-b (density 1),0.1,86028,0,0,0,True,evidence/ios/26-diff.json +ios,first-run,flat a-vs-b (density 1),0.1,86028,0,0,0,True,evidence/ios/30-diff.json +ios,first-run,warm-1,0.1,774252,0,0,0,True,evidence/ios/37-diff.json +ios,first-run,warm-2,0.1,774252,0,0,0,True,evidence/ios/40-diff.json +ios,first-run,warm-3,0.1,774252,0,0,0,True,evidence/ios/43-diff.json +ios,first-run,relaunch-1,0.1,774252,0,0,0,True,evidence/ios/53-diff.json +ios,first-run,gross,0.1,774252,4277,0.55,1,False,evidence/ios/59-diff.json +ios,first-run,gross,0.1,774252,116292,15.02,3,False,evidence/ios/60-diff.json +ios,first-run,revert-1,0.1,774252,0,0,0,True,evidence/ios/63-diff.json +ios,first-run,realistic,0.02,774252,10179,1.31,3,False,evidence/ios/66-diff-t0.02.json +ios,first-run,realistic,0.1,774252,0,0,0,True,evidence/ios/66-diff-t0.1.json +ios,first-run,revert-2,0.1,774252,0,0,0,True,evidence/ios/69-diff.json +ios,first-run,relaunch-1,0.02,774252,0,0,0,True,evidence/ios/6x-diff-relaunch-1-t002.json +ios,first-run,warm-1,0.02,774252,0,0,0,True,evidence/ios/6x-diff-warm-1-t002.json +ios,first-run,warm-2,0.02,774252,0,0,0,True,evidence/ios/6x-diff-warm-2-t002.json +ios,first-run,warm-3,0.02,774252,0,0,0,True,evidence/ios/6x-diff-warm-3-t002.json +android,first-run,relaunch-1,0.01,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.01.json +android,first-run,relaunch-1,0.02,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.02.json +android,first-run,relaunch-1,0.05,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.05.json +android,first-run,relaunch-1,0.1,821760,0,0,0,True,evidence/android/103-diff-relaunch-1-t0.1.json +android,first-run,gross,0.01,821760,265916,32.36,2,False,evidence/android/108-diff-gross-t0.01.json +android,first-run,gross,0.02,821760,208505,25.37,4,False,evidence/android/108-diff-gross-t0.02.json +android,first-run,gross,0.05,821760,118661,14.44,4,False,evidence/android/108-diff-gross-t0.05.json +android,first-run,gross,0.1,821760,65051,7.92,2,False,evidence/android/108-diff-gross-t0.1.json +android,first-run,revert1,0.01,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.01.json +android,first-run,revert1,0.02,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.02.json +android,first-run,revert1,0.05,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.05.json +android,first-run,revert1,0.1,821760,0,0,0,True,evidence/android/111-diff-revert1-t0.1.json +android,first-run,realistic,0.01,821760,14686,1.79,1,False,evidence/android/114-diff-realistic-t0.01.json +android,first-run,realistic,0.02,821760,9336,1.14,1,False,evidence/android/114-diff-realistic-t0.02.json +android,first-run,realistic,0.05,821760,1830,0.22,1,False,evidence/android/114-diff-realistic-t0.05.json +android,first-run,realistic,0.1,821760,0,0,0,True,evidence/android/114-diff-realistic-t0.1.json +android,first-run,revert2,0.01,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.01.json +android,first-run,revert2,0.02,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.02.json +android,first-run,revert2,0.05,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.05.json +android,first-run,revert2,0.1,821760,0,0,0,True,evidence/android/117-diff-revert2-t0.1.json +android,first-run,warm-1,0.01,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.01.json +android,first-run,warm-1,0.02,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.02.json +android,first-run,warm-1,0.05,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.05.json +android,first-run,warm-1,0.1,821760,0,0,0,True,evidence/android/94-diff-warm-1-t0.1.json +android,first-run,warm-2,0.01,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.01.json +android,first-run,warm-2,0.02,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.02.json +android,first-run,warm-2,0.05,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.05.json +android,first-run,warm-2,0.1,821760,0,0,0,True,evidence/android/94-diff-warm-2-t0.1.json +android,first-run,warm-3,0.01,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.01.json +android,first-run,warm-3,0.02,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.02.json +android,first-run,warm-3,0.05,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.05.json +android,first-run,warm-3,0.1,821760,0,0,0,True,evidence/android/94-diff-warm-3-t0.1.json +ios,re-measured,gross,0.02,774252,116292,15.02,3,False,evidence/sensitivity/ios/gross-diff-t0.02.json +ios,re-measured,gross,0.1,774252,4277,0.55,1,False,evidence/sensitivity/ios/gross-diff-t0.1.json +ios,re-measured,realistic-1,0.02,774252,10179,1.31,3,False,evidence/sensitivity/ios/realistic-1-diff-t0.02.json +ios,re-measured,realistic-1,0.1,774252,0,0,0,True,evidence/sensitivity/ios/realistic-1-diff-t0.1.json +ios,re-measured,realistic-2,0.02,774252,10179,1.31,3,False,evidence/sensitivity/ios/realistic-2-diff-t0.02.json +ios,re-measured,realistic-2,0.1,774252,0,0,0,True,evidence/sensitivity/ios/realistic-2-diff-t0.1.json +ios,re-measured,realistic-3,0.02,774252,10179,1.31,3,False,evidence/sensitivity/ios/realistic-3-diff-t0.02.json +ios,re-measured,realistic-3,0.1,774252,0,0,0,True,evidence/sensitivity/ios/realistic-3-diff-t0.1.json +ios,re-measured,reverted,0.02,774252,0,0,0,True,evidence/sensitivity/ios/reverted-diff-t0.02.json +ios,re-measured,reverted,0.1,774252,0,0,0,True,evidence/sensitivity/ios/reverted-diff-t0.1.json +android,re-measured,gross,0.02,821760,208505,25.37,4,False,evidence/sensitivity/android/gross-diff-t0.02.json +android,re-measured,gross,0.1,821760,65051,7.92,2,False,evidence/sensitivity/android/gross-diff-t0.1.json +android,re-measured,gross-probe,0.02,821760,0,0,0,True,evidence/sensitivity/android/gross-probe-diff-t0.02.json +android,re-measured,gross-probe,0.1,821760,0,0,0,True,evidence/sensitivity/android/gross-probe-diff-t0.1.json +android,re-measured,realistic-1,0.02,821760,9336,1.14,1,False,evidence/sensitivity/android/realistic-1-diff-t0.02.json +android,re-measured,realistic-1,0.1,821760,0,0,0,True,evidence/sensitivity/android/realistic-1-diff-t0.1.json +android,re-measured,realistic-2,0.02,821760,9336,1.14,1,False,evidence/sensitivity/android/realistic-2-diff-t0.02.json +android,re-measured,realistic-2,0.1,821760,0,0,0,True,evidence/sensitivity/android/realistic-2-diff-t0.1.json +android,re-measured,realistic-3,0.02,821760,9336,1.14,1,False,evidence/sensitivity/android/realistic-3-diff-t0.02.json +android,re-measured,realistic-3,0.1,821760,0,0,0,True,evidence/sensitivity/android/realistic-3-diff-t0.1.json +android,re-measured,reverted,0.02,821760,0,0,0,True,evidence/sensitivity/android/reverted-diff-t0.02.json +android,re-measured,reverted,0.1,821760,0,0,0,True,evidence/sensitivity/android/reverted-diff-t0.1.json +android,re-measured,sanity-before-break,0.02,821760,0,0,0,True,evidence/sensitivity/android/sanity-before-break-diff-t0.02.json +android,re-measured,sanity-before-break,0.1,821760,0,0,0,True,evidence/sensitivity/android/sanity-before-break-diff-t0.1.json