test(grounder): OpenAI-compatible wire contract + local Qwen VLM probe - #356
test(grounder): OpenAI-compatible wire contract + local Qwen VLM probe#356abrichr wants to merge 4 commits into
Conversation
OpenAICompatibleGrounder was only ever tested against a fake client object;
the real httpx transport path had never run in any test. Stand up an
in-process loopback chat-completions server (the same stdlib pattern as
test_effect_verifier_transport.py) and drive the grounder through its
default module-level httpx.post path, pinning:
- the request shape: POST {base_url}/chat/completions, JSON body with the
model id, the screenshot as a byte-for-byte round-tripping data:image/png
URL, and a prompt carrying the target intent and label; bearer auth on the
wire only when an api_key is configured;
- a valid coordinate reply resolves to exactly that point;
- refusing / malformed / null-coordinate / non-JSON / wrong-shape / non-200
replies abstain (None), never a guessed point;
- an unreachable endpoint returns None, never an exception;
- weakest authority: the reply cannot inflate its own confidence or smuggle
extra fields; the grounder rung is the bottom of RUNG_ORDER and
is_below_ocr, so the risk gate refuses it for irreversible steps; the
adapter is declared MAY_EGRESS for the egress gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/probe_local_grounder.py points the unmodified OpenAICompatibleGrounder at a real OpenAI-compatible endpoint (e.g. a local Ollama /v1) on the same dense-EMR surface and DOM ground truth as benchmark/grounding_eval, recording per-target proposals, errors, hits, misses, and abstentions verbatim. UNRUN: the first run attempt was aborted when the host kernel-panicked under concurrent load with local model inference a likely contributor. The script and benchmark/local_grounder_probe/RESULTS.md state the hard operational requirements for the first real run: a dedicated solo run window (nothing else on the machine), never concurrently with builds, and an explicitly small (8B-class or below) model. Env-gated behind OPENADAPT_GROUNDER_BASE_URL / OPENADAPT_GROUNDER_MODEL; without them it exits UNRUN, so CI never touches a model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first hosted real-model run (Together AI, Qwen/Qwen3.5-9B, 2026-08-12) showed the fixed max_tokens=256 makes a hosted reasoning model truncate mid-reasoning with empty content on 11/12 dense-list targets: every one a safe abstain, but a 100% availability loss. Expose the completion budget as a validated constructor parameter (default unchanged at 256). New tests (tests/test_grounder_max_tokens.py) pin the default wire value, verbatim wire transport of an explicit budget, construction-time rejection of non-positive/non-int budgets, and that truncated replies still abstain. Existing contract tests untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend scripts/probe_local_grounder.py: a second distinct dense surface (seed 2, small_dense) for 12 targets across two screenshots, per-call token usage capture via an injected recording HTTP client (adapter unmodified), a spend estimate with a hard --max-spend cap, and a --max-tokens knob. Add scripts/run_hosted_probe.sh: sources the API key from an operator-named env file at runtime; the key is never stored or printed. Results (benchmark/hosted_grounder_probe/): Qwen/Qwen3.5-9B at the adapter default 256 tokens abstains 11/12 (reasoning exhausts the budget; fail-safe held); at 2048 tokens it scores 0/12 hit@40 with median error 1792 px and every wrong proposal on the wrong row -- the July dense-list failure mode reproduced on hosted open weights. Qwen/Qwen3.7-Plus is streaming-only and yields 12/12 clean abstains. Total spend USD 0.025. Together's classic Qwen VL family is dedicated-endpoint only as of 2026-08-12. Hosted weights are not an on-prem deployment; the local run stays a separate scheduled artifact. Public-artifact inventory regenerated for the three results JSONs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The probe now has real-model results, hosted: |
Summary
Closes the oldest open item from the 2026-08-11 strategy review:
OpenAICompatibleGrounderinopenadapt_flow/runtime/grounder.pyhad never been run against anything but a fake client object — the real transport path had never executed in any test.1. CI contract test (always runs, no model)
tests/test_grounder_openai_compatible_contract.py— 18 tests. An in-process loopback HTTP server (same stdlib pattern astests/test_effect_verifier_transport.py) speaks the OpenAI chat-completions shape, and the grounder is driven through its REAL default path (module-levelhttpx.postover TCP), pinning:POST {base_url}/chat/completions, JSON body with the model id, the screenshot as adata:image/png;base64,URL that round-trips byte-for-byte, a prompt carrying the target intent and label; bearer auth on the wire only when an api_key is configured.None) — never a guessed point.None, never raises.GrounderMatchcarries onlypoint/region/confidence; thegrounderrung is the bottom ofRUNG_ORDERandis_below_ocr, so the risk gate refuses it for irreversible steps; the adapter is declaredMAY_EGRESSfor the egress gate.2. Real local-model probe — committed UNRUN
scripts/probe_local_grounder.py+benchmark/local_grounder_probe/RESULTS.md. Points the unmodified adapter at a real OpenAI-compatible endpoint on the same dense-EMR surface and DOM ground truth asbenchmark/grounding_eval, recording hits/misses/abstentions verbatim. UNRUN: the first run attempt was aborted when the host kernel-panicked under concurrent load, with local model inference a likely contributor. The script and RESULTS.md state the hard requirements for the first real run: dedicated solo run window, never concurrent with builds, explicitly small (8B-class) model. Env-gated behindOPENADAPT_GROUNDER_BASE_URL/OPENADAPT_GROUNDER_MODEL— CI never touches a model.No runtime code changed. No
ci.ymlchange.check_release_consistencypasses unchanged (no artifact-suffix files added).Test plan
uv run pytest tests/test_grounder_openai_compatible_contract.py— 18 passeduv run pytest tests/test_byo_grounding_model.py tests/test_ocr_anchor_grounder.py— 29 passed, 1 skippedruff check/ruff format --checkclean on both new filesscripts/probe_local_grounder.pywithout env vars exits 2 with a plain UNRUN message🤖 Generated with Claude Code