test(example): agent-device visual regression PoC for Surface - #5115
Draft
JKobrynski wants to merge 2 commits into
Draft
test(example): agent-device visual regression PoC for Surface#5115JKobrynski wants to merge 2 commits into
JKobrynski wants to merge 2 commits into
Conversation
Proof of concept for screenshot-based visual regression testing using agent-device against the existing example screens, scoped to Surface. - example/visual/run.mjs: one command per platform (iOS simulator, Android emulator) that opens the example app, navigates to the Surface screen, crops the "Elevated surface" / "Flat surface" sections, and diffs them against committed baselines. Exits 1 on regression; --update re-baselines. - example/visual/README.md: results. Noise floor is 0 px on both platforms across warm captures and a relaunch. agent-device's default diff threshold (0.1) misses a realistic one-level shadow regression on both platforms; 0.02 catches it with noise still at 0. - example/visual/env.json: pinned device/OS/density/versions the baselines are valid for. - SurfaceExample: testID + accessible on the two List.Sections so `screenshot --crop-on` can target them. Twelve agent-device dogfooding issues were recorded locally for filing.
- Drop `accessible` from the two List.Sections. With it, iOS collapsed each
section into a single accessibility element (XCUITest tree evidence in
evidence/a11y/). `testID` alone resolves `screenshot --crop-on` on both
platforms with identical rects, so the example app's accessibility tree
is unchanged by the hook.
- Rename the test ids to surface-example-{elevated,flat} so they cannot be
confused with the library defaults removed in #5088 / #5099. Baselines
renamed to __baselines__/<platform>/<story>.png on both platforms.
- Commit the evidence: raw diff JSON behind every table, break diff images,
issues.md, the a11y trees, and the final script runs.
- run.mjs: env.json mismatch and crop-size mismatch are hard failures
(exit 2 / 3) unless --force; --update can create a missing baseline;
always relaunch so the bundle is fresh (Fast Refresh was seen to stop
reaching Android); handle the Expo dev menu and dev launcher; because
the example app persists navigation state, go Back to the list root and
press the Surface row by position, not the header by label; --force-full
on every snapshot; summary.json written on every exit; --out.
- Re-measure the realistic break three times per platform at pre-declared
thresholds: bit-identical (iOS 10,179 px, Android 9,336 px at 0.02;
0 at 0.1) and identical to the original single captures.
- README: status, evidence links, a11y section, re-measurement section,
issues 13-18, updated Running-it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Our component tests assert style props (
toHaveStyle, tree snapshots) and can't tell whether a component actually looks right. This is a proof of concept for screenshot-based visual regression tests, using agent-device against the existing example-app screens, scoped toSurfaceonly since most other components are mid-rework.It answers one question: does agent-device capture deterministically enough to diff
Surfaceagainst a baseline, and does that diff catch a realistic regression?Result: yes on both platforms, with one catch. Noise floor is 0 px on iOS and Android across warm captures and a process relaunch. But agent-device's default diff threshold (0.1) reports a one-level shadow regression (elevation 1 rendered as 2) as a perfect match on both platforms. At
--threshold 0.02it's caught cleanly, on exactly the right card, with noise still at 0. Full numbers inexample/visual/README.md.Changes
example/visual/run.mjs— one command per platform: opens the example app, navigates to Surface, crops the "Elevated surface" / "Flat surface" sections withscreenshot --crop-on, diffs against the committed baselines. Exit 1 on regression,--updateto re-baseline. Spawns the agent-device CLI (the Node client isn't resolvable from thenpxcache without adding a dependency).example/visual/README.md— results, caveats, what wasn't measured.example/visual/env.json— the pinned device/OS/density/versions the baselines are valid for.example/visual/__baselines__/{ios,android}/— four PNGs.example/src/Examples/SurfaceExample.tsx—testID+accessibleon the twoList.Sections so the crop can target them. Only app change; no library code touched.Related issue
None. Follows the discussion about replacing style-prop assertions with visual tests.
Screenshots / Videos
Baselines are committed under
example/visual/__baselines__/. The diff images from the deliberate breaks (a red ring around only the Elevation 1 card) are in the localartifacts/folder, not committed — happy to attach them here if useful.Test plan
Needs a booted iPhone 17 Pro simulator (iOS 26.5) or the
Pixel_10_ProAVD with the example app installed, Metro running, and agent-device sessions as described in the README's "Running it" section.Expect
changed=0 → PASSfor both stories. To see it fail: changeshadow(elevation, …)toshadow(elevation === 1 ? 2 : elevation, …)in the iOS branch ofSurface.tsx, let Fast Refresh apply, run again — expectsurface-elevatedto FAIL with ~10k changed px.Out of scope
Web (agent-device refuses
--crop-onthere, so it's a different loop), CI, other components, dark theme, cross-platform comparison. Twelve agent-device dogfooding issues were recorded locally and will be filed separately.