Skip to content

fix(scroll): keep the swipe above the keyboard, refuse when it cannot - #2503

Open
thymikee wants to merge 4 commits into
mainfrom
fix/scroll-keyboard-viewport-2500
Open

fix(scroll): keep the swipe above the keyboard, refuse when it cannot#2503
thymikee wants to merge 4 commits into
mainfrom
fix/scroll-keyboard-viewport-2500

Conversation

@thymikee

@thymikee thymikee commented Sep 11, 2026

Copy link
Copy Markdown
Member

Summary

A directional scroll planned a viewport-center-symmetric swipe, so a focused field put the swipe's lower endpoint under the keyboard: the gesture hit keys, the surface never moved, and the edge loop reported a stuck container. The runner now reads its own live keyboard frame, clips the swipe band, and reports keyboardAvoided / keyboardMinY; it refuses with scroll_keyboard_occludes_surface when the keys leave under 15% of the band, and never dismisses the keyboard.

Closes #2500. The Android half lands on top of this in #2514, so this PR stays inside the repository's PR budget.

The clip gate asks whether the keyboard reaches the line the swipe runs along, not what share of the viewport's width the keyboard covers: a vertical swipe travels the viewport's center line, and a floating keyboard straddling that line blocks it however narrow it is. The rule lives in contracts/scroll-gesture behind a golden table asserted by both vitest and an XCTest, so iOS/Android drift turns CI red without a simulator. The superseded scroll keyboard path was deleted, not guarded.

main now carries #2499, so both halves are in: the clip stops the wasted gesture, and #2499's no-progress stop ends the loop.

Review pass

The width-fraction gate is gone from both languages, and the reported case is a shared fixture row (centred floating keyboard under half the viewport width sits in the swipe's path: 834x1112 viewport, 320x332 keyboard centred at x=417, expected keyboardMinY: 780). The Swift test asserts the same row against the same JSON.

Two things found while checking for reuse, both left alone deliberately:

  • frameAvoidingKeyboard (RunnerTests+Interaction.swift:789) still gates on intersection.width / frame.width >= 0.5, i.e. the same defect for tap/drag reference frames. Fixing it changes the tap family, so it belongs in its own change; flagged rather than bundled.
  • keyboardAvoidingDragPoints (same file, :702) keeps a local padding: Double = 12 that names the same fact as this PR's accessoryAllowance, and fails open where scroll refuses. Sharing the constant means editing the drag rule, and the two failure policies differ on purpose: a drag that cannot fit still wants to happen, a scroll entirely under the keys does not.

Size trims: the policy folded into contracts/scroll-gesture rather than a new module (no eager-closure edge to fund); the runner wire code moved to runner-contract.ts instead of coupling the shared fixture to Apple vocabulary; dropped a seeded 200-run property test, a bespoke literalTrueSchema, and a readAndroidGestureViewportReading re-export nothing outside its package consumes. Reused visibleKeyboardFrame, booleanSchema, and the iOS lane's existing -only-testing list.

Validation

pnpm check green at faa75e7860 (exit 0; xctest selection: 0 reachable by no lane).

iOS, iPhone 17 Pro, Form screen, notes field focused:

  • scroll top --jsonreferenceHeight: 571 of 874, keyboardAvoided: true, keyboardMinY: 583, swipe y2=472.
  • First scroll after runner relaunch logs axHealth=unknown keyboardPolicy=requiredWhenAvailable decision=avoided; the old whenAccessibilityHealthy gate skipped the probe there.
  • No keyboard → no evidence keys, referenceHeight: 874, decision=noKeyboard.

iOS, iPadOS 26 windowed Safari (address bar focused): referenceWidth: 834, referenceHeight: 913, keyboardAvoided: true, keyboardMinY: 925. The window is 913pt tall inside a 1219pt display, so this is the clip reading the app window rather than the display.

Unreproduced live: the occluded refusal. The band above the keys cannot fall under 15% on an iPhone (583 of 874 = 65%), and iPadOS 26 keeps an app window clear of the keyboard (window bottom 913, keyboard top 925), so reaching it needs a window resized to roughly the keyboard's own height by dragging it from the host desktop; the macOS desktop session timed out here before the drag. It is covered on both sides by the parity table (adjustPan window with only 88px of 900px left above the IME is refused) and by the platform tests that assert no swipe is planned when the keyboard owns the window.

Size

17 files, 980 gross lines (under the 1,000 budget), 4 commits, one command family. The Android half that used to sit here (helper IME read, input-actions clip, their tests, 463 lines) is #2514, stacked on this branch; its behaviour is the same rule against the same table rows, which land here.

Unrelated failures: the mutation-model ownership assertion reproduces at the merge-base with this diff reverted, and the iOS smoke lab-screen timeout is the reported pre-existing one. check:production-exports passes.

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
PR Preview Action v1.8.1

QR code for preview link

🚀 View preview at
https://callstack.github.io/agent-device/pr-preview/pr-2503/

Built to branch gh-pages at 2026-09-12 12:19 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.50 MB 4.50 MB +7.2 kB
Package (unpacked) 4.50 MB 4.50 MB +7.2 kB
Package (download) 1.32 MB 1.33 MB +2.1 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 23.0 ms 23.8 ms +0.8 ms
CLI --help 64.2 ms 66.5 ms +2.3 ms

@thymikee

thymikee commented Sep 11, 2026

Copy link
Copy Markdown
Member Author

The clipping rule still allows swipes into a floating keyboard at 0fb3f39. Both clipScrollViewportAboveKeyboard and its Swift counterpart ignore keyboards covering less than half the viewport width, but vertical scrolling uses the viewport's center line. A centered 320px keyboard in an 834px viewport covers that line despite being only 38% wide. Check whether the planned swipe intersects the keyboard, and add that centered case to the shared fixture.

Required live evidence is also incomplete: please exercise Android with an adjustPan activity and the iOS scroll_keyboard_occludes_surface refusal, including confirmation that no gesture was dispatched. The reported iOS clipping run and unit parity table do not cover those paths.

The change has 1,561 gross lines and adds 8.2 kB unpacked. Please split it below the repository's 1,000-line PR budget and explain why the smaller owning change is insufficient, or obtain an explicit budget exception. The iOS smoke failed while waiting for the Automation lab screen, before exercising keyboard clipping; that appears unrelated. The separately reported mutation-model failure was reproduced at the base and also appears unrelated.

@thymikee
thymikee force-pushed the fix/scroll-keyboard-viewport-2500 branch from 0fb3f39 to c7f4bd2 Compare September 12, 2026 06:16
@thymikee

Copy link
Copy Markdown
Member Author

Addressed at c7f4bd2b7b.

Centre-line gate. The width fraction is gone from clipScrollViewportAboveKeyboard and its Swift twin; the gate is now swipeCenterX inside the keyboard's x-range, which is what a vertical swipe actually travels. Your case is a shared fixture row, centred floating keyboard under half the viewport width sits in the swipe's path (834x1112 viewport, 320x332 keyboard centred at 417, expected keyboardMinY: 780), asserted by vitest and by testScrollViewportKeyboardClipMatchesGoldenParityTable against the same JSON. Two near-neighbours kept out of scope and named in the body: frameAvoidingKeyboard (RunnerTests+Interaction.swift:789) still uses intersection.width / frame.width >= 0.5 for tap/drag reference frames, and keyboardAvoidingDragPoints (:702) hard-codes the same 12pt allowance and fails open where scroll refuses.

Android adjustPan, live. Pixel 7 emulator, Contacts editor (sim={adjust=pan} in dumpsys window windows), LatinIME bound: helper reports the IME window [0,1517][1080,2400] against a 1080x2400 app window, and scroll down --json returns referenceHeight: 1505, keyboardAvoided: true, keyboardMinY: 1517, swipe 1205 -> 301 — it used to swipe 1920 -> 480 into the keys. Two things only came out on device: UiAutomation.getWindows() returns an empty list until interactive window retrieval is on (c7f4bd2b7b), and open re-activates agent-device's headless imehelper, which draws no keyboard, so a real IME has to be bound after open or there is no IME window to avoid. Also added an iPadOS 26 windowed-Safari reading (834x913 window in a 1219pt display, keyboardAvoided: true, keyboardMinY: 925) to show the clip tracks the app window and not the display.

Still open: the live occluded refusal. Not reachable where I can reach: iPhone leaves 583 of 874 above the keys (65%), and iPadOS 26 keeps a window clear of the keyboard (window bottom 913 vs keyboard top 925), so it needs a window dragged down to about the keyboard's height from the host desktop — the macOS desktop session timed out on this host before I could drag. Table coverage on both sides for now; say the word and I'll keep pushing on the host-side resize.

Size. 1,443 gross (817 production) after the trims listed in the body, so still over 1,000. The clean split is this PR minus Android (~260 lines: helper window read, input-actions clip, their tests) as a follow-up. Prefer the split or an exception?

A directional scroll plans a viewport-center-symmetric swipe, so on iOS and Android alike it aims at
the keyboard whenever a field holds focus. The clip rule that keeps the swipe above the keys has to
agree across the TypeScript and Swift runtimes, so it lives here beside a golden table both of them
assert against rather than in either runtime.

The gate asks whether the keyboard reaches the line the swipe travels, not what share of the
viewport's width the keys cover: a 320pt keyboard centred in an 834pt viewport is 38% of the width and
sits exactly in that path, so a width fraction lets it through. The centred floating keyboard is a
table row now. `ScrollCommandResult` gains `keyboardAvoided` and `keyboardMinY`, present only when a
clip actually happened, because a platform that runs no clip has nothing to report.

The table is the drift detector: a negative-height keyboard case had to be dropped from it while
writing this, because Swift standardizes a negative `CGRect` extent and the two runtimes genuinely
cannot share that row.
…nnot

The runner owns the live keyboard frame, so it does the clip and reports what it left: a scroll
answers with `keyboardAvoided` and `keyboardMinY` beside its plan, and refuses with
`SCROLL_KEYBOARD_OCCLUDES_SURFACE` when the keys leave too little band to swipe in instead of flinging
into them. It never dismisses the keyboard, which would drop focus and mutate state that
session-action provenance does not record.

Scroll's keyboard policy moves to `requiredWhenAvailable`. The probe costs a live AX fetch, but
gating it on a healthy tree left the first scroll of a session swiping under the keys, which is the
failure this is for. Every scroll logs its decision, including the two ways it avoids reading the
keyboard at all.

Scroll no longer shares `frameAvoidingKeyboard`, whose 25% fail-open was a tap-reference-frame rule;
that path is unchanged for its remaining callers.
…reason

The runner's refusal arrives as a runner error code, so it joins the set of codes that keep their
classification in `details.runnerErrorCode` instead of renaming the wire code, and the command maps it
to `scroll_keyboard_occludes_surface` with a hint naming the explicit `keyboard dismiss`. The clip
evidence passes through beside the plan.

`keyboardAvoided` is present only when the swipe was actually clipped. A flat false would read as "no
keyboard" on tvOS and macOS, which share this result normalizer and run no clip, so absence carries
the meaning here.
The parity table only detects drift if both halves run in CI. Two of these three were reachable by no
lane, so the Swift half of the table was a local assertion.
@thymikee

Copy link
Copy Markdown
Member Author

Split per the size ask. This branch is now the contract plus the Apple runner path: 17 files, 980 gross lines, under the 1,000 budget. The Android half (helper IME read, input-actions clip, their tests, 463 lines) is #2514, stacked on this branch.

pnpm check green at faa75e7860 on this branch and on the Android branch. History was rebuilt for the split, so the earlier head is gone; nothing else changed in content — the two branches together differ from the previous head only by the split itself and by one commands.md sentence that now claims iOS only until #2514 lands.

One thing the rebuild surfaced: the previous head also deleted a nine-line comment in interaction-guarantees.ts about the offscreen rescue override's separate-request read (#2448). That was collateral damage in my own edit, not intent, and it is not carried into either branch — main's comment is intact here.

Still open and unchanged from my last note: the live scroll_keyboard_occludes_surface refusal needs an iOS window dragged down to about the keyboard's height from the host desktop (iPhone leaves 65% above the keys, iPadOS 26 holds the window clear at 913 vs 925), and the macOS desktop session timed out here. Table coverage on both sides meanwhile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

scroll flings into the keyboard; keep the swipe in the visible surface and fail closed

1 participant