What happened?
interactive_recorder records e.clientX / e.clientY exactly as the event delivers
them, which is relative to the viewport of the frame the listener fired in. Every other
consumer of those numbers treats them as top-document coordinates:
- frames are captured with
page.screenshot(), so the anchor crops the compiler makes from
x/y are page-space;
PlaywrightBackend._FramePoint is documented as "A top-level point projected into one
concrete document viewport", i.e. replay takes a top-level point and hit-tests down into
the frame chain.
So on a page with no iframes the two spaces coincide and everything works (this is why the
bundled MockMed demo is unaffected); as soon as a click happens inside an iframe the recorded
point is silently wrong by the frame's offset, and nothing warns.
Two consequences:
- Anchors are cropped from the wrong place. Running
compile on such a recording
produces templates that do not contain the target. In our case OCR of the crops returned
text belonging to unrelated parts of the page, and the identity_template came out with
band_len: 0 / tokens: [].
- Two different targets can record as the same point. In the reproduction below, a click
on a top-document button and a click on a button inside a nested iframe 300px away are both
recorded as (80, 35).
Related: anchor.structural.frame_path exists in ir.py and is consumed by
playwright_backend.py, replayer.py, repair/cli.py and bundle_validation.py, but
interactive_recorder.py never sets it — the in-page structuralTarget() emits only
selector / role / name, so interactively-recorded bundles always compile to
frame_path: null and replay cannot re-enter the frame the step was demonstrated in.
Expected: either x/y are top-document coordinates, or the event declares which frame
it is relative to (and frame_path is populated) so consumers can convert.
Steps to reproduce
Save as repro.py and run (pip install "openadapt-flow[browser]"). It serves a page with
a nested iframe, drives the recorder through its own scripted hook, and prints each recorded
point next to Playwright's page-space ground truth:
"""Recorded event coordinates vs page-space truth, on a page with one iframe."""
from __future__ import annotations
import http.server
import json
import threading
from openadapt_flow.interactive_recorder import record_interactive
TOP = """<!doctype html><title>frame repro</title>
<body style="margin:0">
<button id="top" style="position:absolute;left:20px;top:20px;width:120px;height:30px">
top</button>
<iframe id="inner" src="/inner.html" style="position:absolute;left:300px;top:300px;
width:400px;height:200px;border:0"></iframe>
</body>"""
INNER = """<!doctype html><title>inner</title>
<body style="margin:0">
<button id="deep" style="position:absolute;left:20px;top:20px;width:120px;height:30px">
deep</button>
</body>"""
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self) -> None:
body = (INNER if "inner" in self.path else TOP).encode()
self.send_response(200)
self.send_header("Content-Type", "text/html")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *_args) -> None:
pass
def serve() -> int:
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=srv.serve_forever, daemon=True).start()
return srv.server_address[1]
truth: dict[str, tuple[float, float]] = {}
def script(page, pump) -> None:
def settle(n: int = 8) -> None:
for _ in range(n):
pump()
inner = page.frame(url=lambda u: "inner" in u)
assert inner is not None
for name, loc in [("top", page.locator("#top")), ("deep", inner.locator("#deep"))]:
box = loc.bounding_box()
truth[name] = (
round(box["x"] + box["width"] / 2),
round(box["y"] + box["height"] / 2),
)
page.locator("#top").click()
settle()
inner.locator("#deep").click()
settle()
def main() -> None:
out = record_interactive(
f"http://127.0.0.1:{serve()}/top.html",
"rec-upstream-repro",
headless=True,
script=script,
)
events = [
json.loads(line) for line in (out / "events.jsonl").read_text().splitlines()
]
print(f"\n{'target':8s} {'recorded x,y':14s} {'true page x,y':14s} frame_path")
print("-" * 60)
for name, ev in zip(("top", "deep"), events):
print(
f"{name:8s} {str((ev['x'], ev['y'])):14s} {str(truth[name]):14s} "
f"{ev.get('structural', {}).get('frame_path', '<absent>')}"
)
print("\nboth clicks recorded at the same point: "
f"{(events[0]['x'], events[0]['y']) == (events[1]['x'], events[1]['y'])}")
if __name__ == "__main__":
main()Observed on 1.31.0:
target recorded x,y true page x,y frame_path
------------------------------------------------------------
top (80, 35) (80, 35) <absent>
deep (80, 35) (380, 335) <absent>
both clicks recorded at the same point: True
The same thing shows end-to-end with record → compile on any framed app: the compiled
anchors do not contain their targets and every frame_path is null.
field_rect (used for secret redaction) is frame-local in the same way, so a secret field
inside a frame is blacked out at the wrong coordinates and the real field is left visible.
Filed separately per SECURITY.md.
Run report / logs
Not applicable — the divergence is visible in events.jsonl itself, as printed above. Happy
to attach a compiled bundle from a synthetic-data app if useful.
openadapt-flow version
1.31.0 (also reproduced against main @ a2ceac3; interactive_recorder.py is unchanged
since d1b1ced)
OS + Python version
macOS 14.6 (arm64) / Python 3.12
Note on a fix
We are carrying a local patch and are happy to describe it in detail, but the shape of the
real fix is a decision for you, so we are only reporting:
- In-page composition. Walk
window.frameElement up to window.top, accumulating each
frame element's getBoundingClientRect() plus its left/top border and padding, and add that
offset at emit time. We verified this reproduces Playwright's page-space box exactly for
top-document, single-iframe and nested-iframe targets. The same walk yields the frame chain,
which is what frame_path wants. It only works same-origin — frameElement throws across an
origin boundary — so those events have to be refused or handled another way.
- Driver-side alternative.
PlaywrightBackend can already see across origins
(_frame_point, content_frame()), so composing the offset in Python would cover
cross-origin frames too, at the cost of doing it outside the event.
- Compatibility. Changing the meaning of
x/y fixes consumers but reinterprets every
recording made so far; adding separate page-space fields is safe but leaves two coordinate
systems in the format. We have no view on which you prefer — but a recording that does not
declare its coordinate space cannot be consumed safely either way, so some explicit marker
would help downstream tools fail closed instead of guessing.
What happened?
interactive_recorderrecordse.clientX/e.clientYexactly as the event deliversthem, which is relative to the viewport of the frame the listener fired in. Every other
consumer of those numbers treats them as top-document coordinates:
page.screenshot(), so the anchor crops the compiler makes fromx/yare page-space;PlaywrightBackend._FramePointis documented as "A top-level point projected into oneconcrete document viewport", i.e. replay takes a top-level point and hit-tests down into
the frame chain.
So on a page with no iframes the two spaces coincide and everything works (this is why the
bundled MockMed demo is unaffected); as soon as a click happens inside an iframe the recorded
point is silently wrong by the frame's offset, and nothing warns.
Two consequences:
compileon such a recordingproduces templates that do not contain the target. In our case OCR of the crops returned
text belonging to unrelated parts of the page, and the
identity_templatecame out withband_len: 0/tokens: [].on a top-document button and a click on a button inside a nested iframe 300px away are both
recorded as
(80, 35).Related:
anchor.structural.frame_pathexists inir.pyand is consumed byplaywright_backend.py,replayer.py,repair/cli.pyandbundle_validation.py, butinteractive_recorder.pynever sets it — the in-pagestructuralTarget()emits onlyselector/role/name, so interactively-recorded bundles always compile toframe_path: nulland replay cannot re-enter the frame the step was demonstrated in.Expected: either
x/yare top-document coordinates, or the event declares which frameit is relative to (and
frame_pathis populated) so consumers can convert.Steps to reproduce
Save as
repro.pyand run (pip install "openadapt-flow[browser]"). It serves a page witha nested iframe, drives the recorder through its own scripted hook, and prints each recorded
point next to Playwright's page-space ground truth:
Observed on 1.31.0:
The same thing shows end-to-end with
record→compileon any framed app: the compiledanchors do not contain their targets and every
frame_pathisnull.field_rect(used for secret redaction) is frame-local in the same way, so a secret fieldinside a frame is blacked out at the wrong coordinates and the real field is left visible.
Filed separately per SECURITY.md.
Run report / logs
Not applicable — the divergence is visible in
events.jsonlitself, as printed above. Happyto attach a compiled bundle from a synthetic-data app if useful.
openadapt-flow version
1.31.0(also reproduced againstmain@a2ceac3;interactive_recorder.pyis unchangedsince
d1b1ced)OS + Python version
macOS 14.6 (arm64) / Python 3.12
Note on a fix
We are carrying a local patch and are happy to describe it in detail, but the shape of the
real fix is a decision for you, so we are only reporting:
window.frameElementup towindow.top, accumulating eachframe element's
getBoundingClientRect()plus its left/top border and padding, and add thatoffset at emit time. We verified this reproduces Playwright's page-space box exactly for
top-document, single-iframe and nested-iframe targets. The same walk yields the frame chain,
which is what
frame_pathwants. It only works same-origin —frameElementthrows across anorigin boundary — so those events have to be refused or handled another way.
PlaywrightBackendcan already see across origins(
_frame_point,content_frame()), so composing the offset in Python would covercross-origin frames too, at the cost of doing it outside the event.
x/yfixes consumers but reinterprets everyrecording made so far; adding separate page-space fields is safe but leaves two coordinate
systems in the format. We have no view on which you prefer — but a recording that does not
declare its coordinate space cannot be consumed safely either way, so some explicit marker
would help downstream tools fail closed instead of guessing.