record: interactions the injected listeners miss or mis-record (new tabs, select, contenteditable, checkboxes)

[spo0nman]

What happened?

We drove interactive_recorder over a fixture built to exercise each interaction type and
compared what landed in events.jsonl. The listener set (pointerdown, pointerup, click,
contextmenu, input, keydown, wheel, all capture-phase) leaves some gaps that are silent
— the step is simply absent, or present with a value replay cannot act on. Grouping them so
you can triage in one place; happy to split.

Missed entirely (no event at all):

Interaction	Why	Effect
anything done in a new tab	expose_binding and add_init_script are installed on self.page, not on the browser context, so a second page gets no listeners	see below — worst of these
middle / aux click	if (e.button !== 0) return in both pointerdown and click	step lost
hover-revealed menus	no mouseover / mouseenter listener	the click on the revealed item is recorded, but replay never reveals it, so it cannot land
programmatic scroll (scrollTo, scrollIntoView)	only wheel is observed	app-driven scroll position diverges from the recording
native alert / confirm	no dialog handling	dismissal is invisible in the recording

The new-tab hole is the one that hurt us: we clicked a target="_blank" link (recorded,
and pages_after correctly goes 1 → 2), then clicked a button and typed 17 characters in the
second tab, and got zero events. Recording then continued normally in the original tab, so
there is no error and nothing in the recording indicates a gap. Any app with a popup/report
window records as a hole in the middle of the demonstration. Installing the binding and init
script on the context rather than the page would close it.

Recorded, but with a value replay cannot use:

Interaction	Recorded as	Problem
<select>	type carrying the option's value (e.g. v-alpha)	you cannot type into a select, and the value is not the visible label (Label Alpha), so it also cannot be matched on screen. Needs select-option semantics
contenteditable	type with empty text	the typed content is lost entirely; rich-text notes cannot be recorded
checkbox	the correct click plus a phantom type with text: "on"	that is the input listener reading .value of a checkbox; replay would try to type on into it
file input	type with text: "C:\fakepath\patient.txt"	not an actionable step, and it puts the chosen filename in the recording (also flagged privately per SECURITY.md)
double-click	two separate click events	no dblclick; replaying two clicks can double-trigger, and a UI that only responds to dblclick never gets one

One ordering subtlety worth documenting even though it is arguably correct behaviour: a
step's BEFORE frame is the previous settled frame, so when the action itself scrolls the target
into view (Playwright's auto-scroll, or a user's browser doing it) the BEFORE frame is at the
old scroll position and the anchor crop for that step is blank. We found this only because a
downstream check rejected the featureless crop. Scrolling as its own step first avoids it, but
nothing tells the operator that.

Steps to reproduce

Fixture page with: a button whose handler counts dblclick, a select with value/label
differing, a contenteditable div, a checkbox, a file input, a hover-revealed item, a
textarea, and a target="_blank" link. Drive it with record_interactive(..., script=...)
doing: double-click, select_option, type into the contenteditable, check the box, set a file,
click the hover item, type in the textarea, PageDown, click the new-tab link, then act in the
new tab, then act again in the first tab. Compare against events.jsonl.

We can attach the fixture and driver if that is useful — say the word and we will paste both.

Run report / logs

events.jsonl from the run above, abbreviated to the relevant fields:

{"i": 0, "kind": "click", "structural": {"selector": "#dbl"}}          // double-click →
{"i": 1, "kind": "click", "structural": {"selector": "#dbl"}}          // two clicks
{"i": 2, "kind": "type", "text": "v-alpha", "field_label": "sel"}      // <select> value, not label
{"i": 3, "kind": "click", "structural": {"selector": "#ce"}}
{"i": 4, "kind": "type", "text": ""}                                   // contenteditable, content lost
{"i": 5, "kind": "click", "structural": {"selector": "#chk"}}
{"i": 6, "kind": "type", "text": "on", "field_label": "chk"}           // phantom checkbox type
{"i": 7, "kind": "type", "text": "C:\\fakepath\\patient.txt"}          // file input
{"i": 9, "kind": "click", "structural": {"selector": "#hoveritem"}}    // no preceding hover
{"i": 13, "kind": "click", "structural": {"selector": "#newtab"}, "pages_after": 2}
{"i": 14, "kind": "click", "structural": {"selector": "#dbl"}, "pages_before": 2}

Between i: 13 and i: 14 a button click and 17 typed characters happened in the second tab.
Nothing was recorded for them. There is no alert step either (the alert() click at i: 8
is recorded, the dismissal is not).

openadapt-flow version

1.31.0 (also on main @ a2ceac3)

OS + Python version

macOS 14.6 (arm64) / Python 3.12