Skip to content

test(grounder): OpenAI-compatible wire contract + local Qwen VLM probe - #356

Open
abrichr wants to merge 4 commits into
mainfrom
feat/local-grounder-proof
Open

test(grounder): OpenAI-compatible wire contract + local Qwen VLM probe#356
abrichr wants to merge 4 commits into
mainfrom
feat/local-grounder-proof

Conversation

@abrichr

@abrichr abrichr commented Aug 12, 2026

Copy link
Copy Markdown
Member

Summary

Closes the oldest open item from the 2026-08-11 strategy review: OpenAICompatibleGrounder in openadapt_flow/runtime/grounder.py had never been run against anything but a fake client object — the real transport path had never executed in any test.

1. CI contract test (always runs, no model)

tests/test_grounder_openai_compatible_contract.py — 18 tests. An in-process loopback HTTP server (same stdlib pattern as tests/test_effect_verifier_transport.py) speaks the OpenAI chat-completions shape, and the grounder is driven through its REAL default path (module-level httpx.post over TCP), pinning:

  • Request shape: POST {base_url}/chat/completions, JSON body with the model id, the screenshot as a data:image/png;base64, URL that round-trips byte-for-byte, a prompt carrying the target intent and label; bearer auth on the wire only when an api_key is configured.
  • Valid reply resolves to exactly the returned point (string and content-parts-list shapes both).
  • Refusing / malformed / null-coordinate / non-JSON / wrong-shape / non-200 replies abstain (None) — never a guessed point.
  • Unreachable endpoint returns None, never raises.
  • Weakest authority: the reply cannot inflate its own confidence (fixed 0.5) or smuggle extra fields; GrounderMatch carries only point/region/confidence; the grounder rung is the bottom of RUNG_ORDER and is_below_ocr, so the risk gate refuses it for irreversible steps; the adapter is declared MAY_EGRESS for the egress gate.

2. Real local-model probe — committed UNRUN

scripts/probe_local_grounder.py + benchmark/local_grounder_probe/RESULTS.md. Points the unmodified adapter at a real OpenAI-compatible endpoint on the same dense-EMR surface and DOM ground truth as benchmark/grounding_eval, recording hits/misses/abstentions verbatim. UNRUN: the first run attempt was aborted when the host kernel-panicked under concurrent load, with local model inference a likely contributor. The script and RESULTS.md state the hard requirements for the first real run: dedicated solo run window, never concurrent with builds, explicitly small (8B-class) model. Env-gated behind OPENADAPT_GROUNDER_BASE_URL / OPENADAPT_GROUNDER_MODEL — CI never touches a model.

No runtime code changed. No ci.yml change. check_release_consistency passes unchanged (no artifact-suffix files added).

Test plan

  • uv run pytest tests/test_grounder_openai_compatible_contract.py — 18 passed
  • uv run pytest tests/test_byo_grounding_model.py tests/test_ocr_anchor_grounder.py — 29 passed, 1 skipped
  • ruff check / ruff format --check clean on both new files
  • scripts/probe_local_grounder.py without env vars exits 2 with a plain UNRUN message

🤖 Generated with Claude Code

abrichr and others added 4 commits August 12, 2026 12:01
OpenAICompatibleGrounder was only ever tested against a fake client object;
the real httpx transport path had never run in any test. Stand up an
in-process loopback chat-completions server (the same stdlib pattern as
test_effect_verifier_transport.py) and drive the grounder through its
default module-level httpx.post path, pinning:

- the request shape: POST {base_url}/chat/completions, JSON body with the
  model id, the screenshot as a byte-for-byte round-tripping data:image/png
  URL, and a prompt carrying the target intent and label; bearer auth on the
  wire only when an api_key is configured;
- a valid coordinate reply resolves to exactly that point;
- refusing / malformed / null-coordinate / non-JSON / wrong-shape / non-200
  replies abstain (None), never a guessed point;
- an unreachable endpoint returns None, never an exception;
- weakest authority: the reply cannot inflate its own confidence or smuggle
  extra fields; the grounder rung is the bottom of RUNG_ORDER and
  is_below_ocr, so the risk gate refuses it for irreversible steps; the
  adapter is declared MAY_EGRESS for the egress gate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/probe_local_grounder.py points the unmodified
OpenAICompatibleGrounder at a real OpenAI-compatible endpoint (e.g. a local
Ollama /v1) on the same dense-EMR surface and DOM ground truth as
benchmark/grounding_eval, recording per-target proposals, errors, hits,
misses, and abstentions verbatim.

UNRUN: the first run attempt was aborted when the host kernel-panicked under
concurrent load with local model inference a likely contributor. The script
and benchmark/local_grounder_probe/RESULTS.md state the hard operational
requirements for the first real run: a dedicated solo run window (nothing
else on the machine), never concurrently with builds, and an explicitly
small (8B-class or below) model.

Env-gated behind OPENADAPT_GROUNDER_BASE_URL / OPENADAPT_GROUNDER_MODEL;
without them it exits UNRUN, so CI never touches a model.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first hosted real-model run (Together AI, Qwen/Qwen3.5-9B, 2026-08-12)
showed the fixed max_tokens=256 makes a hosted reasoning model truncate
mid-reasoning with empty content on 11/12 dense-list targets: every one a
safe abstain, but a 100% availability loss. Expose the completion budget as
a validated constructor parameter (default unchanged at 256).

New tests (tests/test_grounder_max_tokens.py) pin the default wire value,
verbatim wire transport of an explicit budget, construction-time rejection
of non-positive/non-int budgets, and that truncated replies still abstain.
Existing contract tests untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend scripts/probe_local_grounder.py: a second distinct dense surface
(seed 2, small_dense) for 12 targets across two screenshots, per-call token
usage capture via an injected recording HTTP client (adapter unmodified),
a spend estimate with a hard --max-spend cap, and a --max-tokens knob.
Add scripts/run_hosted_probe.sh: sources the API key from an operator-named
env file at runtime; the key is never stored or printed.

Results (benchmark/hosted_grounder_probe/): Qwen/Qwen3.5-9B at the adapter
default 256 tokens abstains 11/12 (reasoning exhausts the budget; fail-safe
held); at 2048 tokens it scores 0/12 hit@40 with median error 1792 px and
every wrong proposal on the wrong row -- the July dense-list failure mode
reproduced on hosted open weights. Qwen/Qwen3.7-Plus is streaming-only and
yields 12/12 clean abstains. Total spend USD 0.025. Together's classic Qwen
VL family is dedicated-endpoint only as of 2026-08-12.

Hosted weights are not an on-prem deployment; the local run stays a
separate scheduled artifact. Public-artifact inventory regenerated for the
three results JSONs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@abrichr

abrichr commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

The probe now has real-model results, hosted: benchmark/hosted_grounder_probe/RESULTS.md (Together AI, 2026-08-12). Together's classic Qwen VL family is dedicated-endpoint-only as of today, so the probed pair is the two hosted Qwen models the shipped adapter can reach. Qwen/Qwen3.5-9B (smallest serverless image-capable Qwen, $0.17/$0.25 per 1M) at the adapter's default 256-token budget abstained on 11/12 dense-list targets — the reasoning model exhausts the budget before emitting content, and the fail-safe held (abstain, never a guessed click); this motivated the new validated max_tokens constructor parameter (pinned by tests/test_grounder_max_tokens.py, existing contract tests untouched). At 2048 tokens the honest capability number is 0/12 hit@40, median error 1792 px, with every wrong proposal on the wrong ROW — the July dense-list failure mode reproduced on hosted, unquantized open weights. Qwen/Qwen3.7-Plus (strongest hosted Qwen) is streaming-only and produced 12/12 clean abstains at $0. Total spend $0.025 of a $5 cap. Caveat recorded in RESULTS.md: hosted weights are not an on-prem deployment — this measures model capability and wire-protocol correctness; the local/on-prem run remains the separate scheduled artifact in benchmark/local_grounder_probe/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant