Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## Unreleased

- Changed: a capture that a backend cut at one of its limits now says so in the snapshot's
warnings, on every platform, instead of only setting `truncated: true` in JSON. The text path
had no disclosure at all, so an agent read a screen missing its footer, tab bar, or the items
after a long list as complete — the backends walk the tree in document order, so what falls
off is what comes last, on screen or not. One shared warning renders from the shared flag; the
limit and dimension stay backend-side.
- Changed: the iOS Simulator AX bridge caps a capture at 5000 nodes, up from 1500, the Android
helper's bound. Measured on a synthetic 600-row screen, acquisition time did not move with the
cap (the native read fetches the whole tree; the cap only stops conversion) while the 1500 cut
dropped the screen's on-screen footer.
- Fixed: Android snapshots carry the accessibility `selected` state an app sets on a control, so
`is selected`, a `selected=true` selector, and a Maestro `selected:` qualifier work on Android
(#2462). The helper never serialized the attribute, and the host reads only the helper's XML, so
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,20 @@ function customActionCoverageWarning(verdict: SnapshotQualityVerdict): string[]
return lines;
}

/**
* One disclosure for every backend that reports a cut capture: the iOS Simulator bridge and the
* Android helper stop at a node cap, the XCTest runner and the web provider at their own bounds.
* Every one walks the tree in document order, so what falls off is what comes last — footers,
* tab bars, the items after a long list — even when it is on screen. The fact is the shared
* `truncated` flag; the dimension and limit stay backend-side, so the copy names neither.
*/
export function truncatedCaptureWarning(truncated: boolean | undefined): string[] {
if (truncated !== true) return [];
return [
'This capture was cut at a backend limit, so elements later in the tree — footers, tab bars, items after a long list — may be missing even when they are on screen; their refs and selectors cannot be resolved from this snapshot. Navigate or scroll so fewer elements render and re-run, and use screenshot as visual truth for what is missing.',
];
}

export function recoveredSnapshotQualityWarning(
backend: SnapshotQualityVerdict['backend'],
): string {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import assert from 'node:assert/strict';
import { test } from 'vitest';
import { readSnapshotQualityVerdict } from '../../snapshot-quality-verdict.ts';
import { renderSnapshotQualityWarnings } from './quality-warnings.ts';
import { renderSnapshotQualityWarnings, truncatedCaptureWarning } from './quality-warnings.ts';

const sharedRecoveryReason =
'iOS XCTest snapshot failed while serializing the accessibility tree. Error kAXErrorIllegalArgument getting snapshot for element <AXUIElementRef 0x1>';
Expand Down Expand Up @@ -270,3 +270,14 @@ test('the coverage line is independent of degradation state', () => {
assert.equal(warnings.length, 1);
assert.match(warnings[0] ?? '', /read for 3 of 8 merged elements/);
});

test('a cut capture is disclosed once, from the shared truncated flag alone', () => {
assert.deepEqual(truncatedCaptureWarning(false), []);
assert.deepEqual(truncatedCaptureWarning(undefined), []);
const [warning, ...rest] = truncatedCaptureWarning(true);
assert.deepEqual(rest, []);
assert.match(warning ?? '', /cut at a backend limit/);
assert.match(warning ?? '', /footers, tab bars/);
assert.match(warning ?? '', /refs and selectors cannot be resolved/);
assert.match(warning ?? '', /screenshot/);
});
2 changes: 1 addition & 1 deletion packages/platform-apple/src/snapshot-source/limits.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ const MAXIMUM_NODES = 10_000;
export const DEFAULT_SNAPSHOT_SOURCE_LIMITS: SnapshotSourceLimits = Object.freeze({
maxRequestBytes: 64 * 1024,
maxResponseBytes: 4 * 1024 * 1024,
maxNodes: 1500,
maxNodes: 5000,
maxTraversalDepth: 64,
maxDurationMs: 5_000,
});
Expand Down
31 changes: 30 additions & 1 deletion src/commands/capture/runtime/snapshot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ test('runtime snapshot renders the structured quality verdict and skips legacy d

const result = await device.capture.snapshot({ session: 'default' });

assert.equal(result.warnings?.length, 2);
assert.equal(result.warnings?.length, 3);
assert.match(
String(result.warnings?.[0]),
/Detected an overly complex or slow accessibility tree/,
Expand All @@ -359,6 +359,8 @@ test('runtime snapshot renders the structured quality verdict and skips legacy d
assert.match(String(result.warnings?.[0]), /It is OK to continue/);
assert.match(String(result.warnings?.[0]), /snapshotQuality\.reason/);
assert.match(String(result.warnings?.[1]), /@e2 \[Other\] merges many labels/);
// The fixture is a cut capture; the shared disclosure follows the verdict's own warnings.
assert.match(String(result.warnings?.[2]), /cut at a backend limit/);
assert.deepEqual(result.snapshotQuality?.state, 'recovered');
});

Expand Down Expand Up @@ -703,3 +705,30 @@ function assertReactNativeOverlayWarning(warnings: string[] | undefined) {
assert.match(warnings[0] ?? '', /agent-device react-native dismiss-overlay/);
assert.match(warnings[0] ?? '', /verifies the overlay is gone/);
}

test('runtime snapshot discloses a cut capture the same way on every backend', async () => {
for (const backend of ['android', 'xctest'] as const) {
const device = createSnapshotOnlyDevice({
nodes: [{ ref: 'e1', index: 0, depth: 0, type: 'Window', label: 'Home' }],
truncated: true,
backend,
});

const result = await device.capture.snapshot({ session: 'default' });

assert.equal(result.truncated, true, backend);
const cut = (result.warnings ?? []).filter((warning) => /cut at a backend limit/.test(warning));
assert.equal(cut.length, 1, `${backend}: ${JSON.stringify(result.warnings)}`);
}

const complete = createSnapshotOnlyDevice({
nodes: [{ ref: 'e1', index: 0, depth: 0, type: 'Window', label: 'Home' }],
truncated: false,
backend: 'android',
});
const result = await complete.capture.snapshot({ session: 'default' });
assert.equal(
(result.warnings ?? []).some((warning) => /cut at a backend limit/.test(warning)),
false,
);
});
6 changes: 5 additions & 1 deletion src/commands/capture/runtime/snapshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@ import {
buildSnapshotDiff,
countSnapshotComparableLines,
} from '@agent-device/capture-kit/snapshot-diff';
import { renderSnapshotQualityWarnings } from '@agent-device/capture-kit/quality-warnings';
import {
renderSnapshotQualityWarnings,
truncatedCaptureWarning,
} from '@agent-device/capture-kit/quality-warnings';
import { buildSnapshotVisibility } from '@agent-device/capture-kit/snapshot-visibility';
import { ANDROID_SYSTEM_SURFACE_DISCLOSURE } from '@agent-device/contracts/android-system-surface-disclosure';
import { formatReactNativeOverlayWarning } from '../../react-native/overlay.ts';
Expand Down Expand Up @@ -247,6 +250,7 @@ function buildSnapshotWarnings(params: {
...renderSnapshotQualityWarnings(params.annotations.quality, params.snapshot.nodes),
);
}
warnings.push(...truncatedCaptureWarning(snapshotTruncationForResult(params.snapshot)));
warnings.push(...buildEmptyAndroidInteractiveWarnings(params));
if (!params.annotations.quality) {
// Legacy runners without a structured verdict keep the old daemon-side heuristics.
Expand Down
6 changes: 6 additions & 0 deletions website/docs/docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,12 @@ agent-device get attrs @e1
Android `--raw` is the acquired tree: it also keeps nodes Android marks invisible and stale
application windows. The helper does not report `checked`/`checkable` state, and it caps
captures at 5000 nodes before any `--scope` applies (`truncated: true`).
- `truncated: true` means the backend cut the capture at one of its limits — the Android helper
and the iOS Simulator AX bridge at 5000 nodes, the XCTest runner and the web provider at their
own bounds. Every backend walks the tree in document order, so what falls off is what comes
last: footers, tab bars, items after a long list, even when on screen. The snapshot carries a
warning that says so; navigate or scroll so fewer elements render and re-run, and use
`screenshot` as visual truth for the rest.
- `--scope <text|@ref>` returns the subtree of the first node in document order whose label, value,
or identifier contains the scope text (case-insensitive) and whose subtree still has content in
the requested projection, re-rooted at depth 0; no match returns an empty snapshot rather than the
Expand Down
Loading