Skip to content

feat(diff): accept JPEG inputs for screenshot comparison - #2718

Open
thymikee wants to merge 3 commits into
mainfrom
feat/diff-screenshot-jpeg-input
Open

thymikee wants to merge 3 commits into
mainfrom
feat/diff-screenshot-jpeg-input

Conversation

@thymikee

Copy link
Copy Markdown
Member

Summary

diff screenshot decoded both inputs as PNG, so a capture exported by another tool had to be converted first and a HarmonyOS artifact — which the device serves as JPEG under whatever name the command was given — could never be compared. Each input is now decoded from its own bytes: the container is sniffed, so a .png name holding JPEG decodes as JPEG.

png-transcode.ts became screenshot-image.ts, the one owner of container sniffing for both the decode path and the provider transcode path, and the PNG worker gained a decode-image job that answers pixels instead of PNG bytes, so a full-resolution decode never blocks the daemon event loop. The --out diff image stays PNG, as do the crop, overlay, and resize passes that rewrite a screenshot in place.

13 files, ~80 net production lines. No new flag, no schema or wire change.

agent-device diff screenshot --baseline baseline.jpg --out diff.png
agent-device diff screenshot --baseline baseline.png harmony-capture.jpg --threshold 0.2

Validation

pnpm check:affected --run passed (all runnable checks) at 46b0079daf, after pnpm check:quick. 8d3a8719a9 only restores gate-generated .agent-device/test-artifacts files to their origin/main content; no source changed.

New coverage: worker↔sync decode parity for both containers, decode-image transfer ownership, a JPEG stored under a .png name comparing exactly at --threshold 0, decoded dimensions on a JPEG, corrupt-JPEG and neither-container refusals with their typed details, and the diff artifact pinned to a PNG signature. The screenshot-diff suite now runs the shipped worker client instead of a module mock.

Unresolved risks, both pre-existing and deliberately not folded in here:

  • packages/platform-harmonyos/src/screenshot.ts:15 still writes JPEG bytes into a .png artifact, so --scale and --overlay-refs fail on that platform. The provider-side transcode Limrun got in fix(limrun): write the iOS screenshot as PNG instead of Limrun's JPEG bytes #2645 is the owning fix; this PR makes those artifacts diffable meanwhile.
  • A hostile input file is fully decoded before maxImagePixels is consulted (already true for PNG); jpeg-js self-caps at 100 MP / 512 MB.

Both diff screenshot inputs had to be PNG, so a capture from another tool had to be converted first and a HarmonyOS artifact, which the platform serves as JPEG, never compared. Each input is now decoded from its own bytes.

png-transcode.ts became screenshot-image.ts, the one owner of container sniffing for both the decode and the provider transcode path, and the PNG worker gained a decode-image job that answers pixels instead of PNG bytes.
Drops the screenshot-diff module mock and the package export that existed only to feed it, so the JPEG cases run the same worker client production uses; pins the diff artifact container and the decode-image transfer arm, and corrects two doc claims the first commit overreached on: HarmonyOS still serves JPEG, and a stored JPEG compares exactly against itself at threshold 0.
The gate rewrites those timing and result artifacts on every run; they were swept in by a broad stage, not changed by this feature.
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-2718/

Built to branch gh-pages at 2026-09-20 20:50 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.67 MB 4.67 MB +966 B
Package (unpacked) 4.67 MB 4.67 MB +966 B
Package (download) 1.39 MB 1.39 MB +483 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 20.5 ms 19.5 ms -1.0 ms
CLI --help 54.3 ms 56.5 ms +2.2 ms

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 8d3a871. This looks ready to merge. CI is green across all 20 checks, and I didn't find anything that blocks it.

I traced the JPEG decode and re-encode paths manually rather than running the suite, relying on CI green plus reading the code against the test assertions.

Not blocking: transcodeScreenshotToPng narrows the try/catch around PNG.sync.write compared to the old jpegToPng (https://github.com/callstack/agent-device/blob/8d3a871/packages/capture-kit/src/screenshot-image.ts#L43), so a re-encode failure would surface as a raw Error instead of the typed COMMAND_FAILED — worth a look, but fine to take or leave as a follow-up.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant