OpenAdaptAI · abrichr · Aug 12, 2026 · Aug 12, 2026
@@ -0,0 +1,47 @@
+# Qwen grounder endpoint — verification results
+
+Status: **NOT DEPLOYED — blocked on the auth secret.**
+
+Preflight on 2026-08-12 found Modal auth present (`~/.modal.toml`, profile
+`abrichr`) but no Modal secret named `qwen-endpoint-token`. Per policy the
+agent does not create or handle the token; the founder command is in
+`RUNBOOK.md` section 0 (and in `NEEDS_YOU.md`). Once the secret exists, run
+RUNBOOK sections 1–2 and fill this file in.
+
+## Endpoint
+
+| What | Value |
+|---|---|
+| Deployed | no (pending secret) |
+| Model | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5`, bf16 (no quantization) |
+| GPU | A10G 24 GB |
+| Cold-start latency | _(measure: RUNBOOK 2.2)_ |
+| Warm request latency | _(measure: RUNBOOK 2.3)_ |
+| Unauthed request | _(expect 401: RUNBOOK 2.1)_ |
+| Scaled to zero after idle | _(expect yes: RUNBOOK 2.4)_ |
+| Verification GPU spend | _(cap $5.00; expected ~$0.20–0.30)_ |
+
+## Grounder smoke (5 requests, 2 committed flow fixtures)
+
+Produced by `smoke_grounder.py` through openadapt-flow's real
+`OpenAICompatibleGrounder`. Fixtures: `benchmark/dense_surface/
+record_seed1.png` (2240x3702) and `benchmark/dense_surface/
+replay_native_arial_seed1.png` (1120x1858) from the openadapt-flow repo.
+
+| fixture | intent | verdict | point | latency_s | est_cost_usd |
+|---|---|---|---|---|---|
+| _(pending deploy)_ | | | | | |
+
+Summary: _(hit / miss / abstain counts, median latency, cost per request)_
+
+Interpretation notes, fixed in advance:
+
+* This is a smoke (wire works end-to-end), not an accuracy claim. The full
+  accuracy probe runs against Together in a sibling effort — compare cost
+  and hit rate there before choosing a default grounding backend.
+* Qwen VL models answer in the coordinate frame of the server-side
+  (possibly resized) image; a systematic offset on the hi-dpi fixture is
+  recorded as `miss`, not tuned away.
+* `abstain` (grounder returns None) is the fail-safe path working: any
+  transport error, non-200, or `{"x": null}` reply must halt the ladder,
+  never click.
@@ -0,0 +1,136 @@
+# Qwen grounder endpoint — deploy / verify / teardown runbook
+
+Self-hosted Qwen VL endpoint on Modal for openadapt-flow's
+`OpenAICompatibleGrounder`. OpenAI-compatible, bearer-token auth,
+scale-to-zero (idle GPU dies within 120 s, no warm pool).
+
+| What | Value |
+|---|---|
+| App | `qwen-grounder-endpoint` (`qwen_endpoint/app.py`) |
+| Model | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5` |
+| Served model id | `qwen2.5-vl-7b-instruct` |
+| Serving | vLLM `0.10.1.1`, bf16, `--max-model-len 16384` |
+| GPU | one A10G (24 GB) |
+| URL | `https://<workspace>--qwen-grounder-endpoint-serve.modal.run/v1` (printed by `modal deploy`; workspace `abrichr`) |
+| Auth | Modal secret `qwen-endpoint-token`, key `TOKEN`; unauthed requests get HTTP 401 from vLLM `--api-key` |
+| Scale-to-zero | `scaledown_window=120`, no `min_containers` (guarded by `tests/test_qwen_endpoint.py`) |
+
+## 0. One-time: create the auth secret (FOUNDER ONLY)
+
+Agents never create, read, or copy this token. The founder runs, once:
+
+```bash
+python3 -c "import secrets;print(secrets.token_urlsafe(32))" | { read t; modal secret create qwen-endpoint-token TOKEN="$t"; security add-generic-password -s oa-qwen-endpoint-token -a modal -w "$t"; }
+```
+
+This mints the token, stores it as the Modal secret `qwen-endpoint-token`
+(read only by the serving container) and as the macOS Keychain item
+`oa-qwen-endpoint-token` (read only by `with_token.sh` at client runtime).
+The value itself is never displayed.
+
+Everything below assumes `modal secret list` shows `qwen-endpoint-token`.
+
+## 1. Deploy
+
+```bash
+cd /Users/abrichr/oa/src/openadapt-ops
+modal deploy qwen_endpoint/app.py
+```
+
+`modal deploy` prints the web URL. The FIRST-ever cold start additionally
+downloads ~16 GB of weights into the `qwen-grounder-endpoint-hf-cache`
+volume (one-time, a few minutes); later cold starts load from the volume.
+
+## 2. Verify
+
+All authed calls go through the launcher so the token stays in the Keychain:
+
+```bash
+cd /Users/abrichr/oa/src/openadapt-ops/qwen_endpoint
+BASE=https://abrichr--qwen-grounder-endpoint-serve.modal.run/v1
+```
+
+1. **Unauthed => 401** (no launcher, no token):
+
+   ```bash
+   curl -s -o /dev/null -w '%{http_code}\n' "$BASE/models"        # expect 401
+   curl -s -o /dev/null -w '%{http_code}\n' -X POST "$BASE/chat/completions" \
+     -H 'Content-Type: application/json' \
+     -d '{"model":"qwen2.5-vl-7b-instruct","messages":[{"role":"user","content":"hi"}]}'   # expect 401
+   ```
+
+2. **Cold-start latency** (first authed call after idle; time it):
+
+   ```bash
+   time ./with_token.sh sh -c 'curl -s -H "Authorization: Bearer $OPENADAPT_FLOW_GROUNDING_API_KEY" "$0/models"' "$BASE"
+   ```
+
+3. **Warm latency + well-formed vision output** — repeat the same call, then
+   run the grounder smoke (5 image requests through the real flow client):
+
+   ```bash
+   ./with_token.sh uv run --project /Users/abrichr/oa/src/openadapt-flow \
+     python3 smoke_grounder.py --base-url "$BASE" \
+     --flow-repo /Users/abrichr/oa/src/openadapt-flow --gpu-usd-per-hour 1.10
+   ```
+
+   Paste the printed table into `RESULTS.md`.
+
+4. **Scale-to-zero**: wait >2 minutes with no traffic, then:
+
+   ```bash
+   modal app list          # qwen-grounder-endpoint shows 0 running containers
+   modal container list    # no container for the app
+   ```
+
+## 3. Teardown
+
+```bash
+modal app stop qwen-grounder-endpoint     # removes the deployment + URL
+# optional, reclaims ~16 GB of weight-cache storage:
+modal volume delete qwen-grounder-endpoint-hf-cache
+modal volume delete qwen-grounder-endpoint-vllm-cache
+```
+
+Founder-only, if retiring the endpoint for good: `modal secret delete
+qwen-endpoint-token` and `security delete-generic-password -s
+oa-qwen-endpoint-token -a modal`.
+
+## 4. Cost math
+
+Rates are the Modal list prices as of 2026-08 — re-check
+`https://modal.com/pricing` before trusting a forecast.
+
+| Item | Math | Cost |
+|---|---|---|
+| A10G GPU-second | ~$1.10/h | ~$0.000306/s |
+| Cold start (volume-cached weights) | ~120–240 s | $0.04–0.08 |
+| First-ever cold start (HF download) | +~180 s | +~$0.06 |
+| Warm grounding request | ~2–6 s | <$0.002 |
+| Idle tail after a burst | exactly 120 s | $0.037 |
+| **Idle deployment (steady state)** | 0 GPU-s | **$0.00 GPU** + volume storage (~16 GB, order of $0.50/mo; confirm in dashboard) |
+| Full verification pass (steps 1–4 above) | ~10–15 min GPU | **~$0.20–0.30** (cap: $5.00) |
+
+Example month — 500 grounding calls in 50 bursts: 50 × (cold 180 s + calls
+~30 s + idle 120 s) ≈ 4.6 GPU-hours ≈ **$5/mo**. The same calls against a
+hosted per-token API are the sibling Together effort's numbers; compare
+before committing either way.
+
+## 5. Pinned-version upgrade path (do NOT bump casually)
+
+* Any bump of `MODEL_REVISION`, `VLLM_VERSION`, or the model itself requires
+  a full section-2 verification pass in the same PR, with measured numbers.
+* Designated next model: `Qwen/Qwen3-VL-8B-Instruct` @
+  `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b` — needs `vllm>=0.11.0`, and at
+  ~17 GB bf16 it is tight on 24 GB: expect to lower `--max-model-len`, or
+  use the official FP8 checkpoint on an L4/Ada GPU (A10G/Ampere has no
+  native FP8).
+* `tests/test_qwen_endpoint.py` pins the safety invariants (auth secret
+  name, 120 s scaledown, no warm pool, pinned revision); it must keep
+  passing untouched.
+
+## 6. Client wiring
+
+`deployment.snippet.yaml` in this directory is the operator-facing flow
+config. Run flow under the launcher so the env var named by `api_key_env`
+exists: `./with_token.sh openadapt-flow run --config deployment.yaml ...`.
@@ -0,0 +1,131 @@
+"""Modal app: Qwen VL grounding endpoint (OpenAI-compatible, scale-to-zero).
+
+Serves ``Qwen/Qwen2.5-VL-7B-Instruct`` (pinned revision) via vLLM behind an
+OpenAI-compatible ``/v1/chat/completions`` route, sized for one 24 GB GPU.
+
+Design constraints (enforced by ``tests/test_qwen_endpoint.py``):
+
+* Scale-to-zero: ``scaledown_window`` is at most 120 s; no container floor,
+  no warm pool. An idle deployment costs $0 in GPU time.
+* Auth: every request must carry ``Authorization: Bearer <token>`` where the
+  token comes from the Modal secret ``qwen-endpoint-token`` (key ``TOKEN``).
+  vLLM's ``--api-key`` flag rejects anything else with HTTP 401. The token is
+  read INSIDE the container from the secret; it never appears in this repo,
+  in logs, or in the deploy output.
+* Pinned model: the exact HuggingFace revision is pinned below so a re-deploy
+  serves byte-identical weights.
+
+Deploy / verify / teardown: see ``qwen_endpoint/RUNBOOK.md``.
+
+Client wiring: openadapt-flow's ``OpenAICompatibleGrounder`` points at
+``https://<workspace>--qwen-grounder-endpoint-serve.modal.run/v1`` — see
+``qwen_endpoint/deployment.snippet.yaml``.
+"""
+
+import subprocess
+
+import modal
+
+APP_NAME = "qwen-grounder-endpoint"
+
+# -- model pin ---------------------------------------------------------------
+# Qwen2.5-VL-7B-Instruct: the largest Qwen VL known to serve reliably on one
+# 24 GB GPU under vLLM in bf16 (~15.5 GB weights + KV cache at 16k context).
+# Revision = HF main as read on 2026-08-12 (last modified 2025-04-06).
+#
+# Qwen3-VL-8B-Instruct (revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b) is
+# the designated upgrade once a deploy can be GPU-verified: it needs
+# vllm>=0.11.0 and is tighter on 24 GB (~17 GB bf16 weights); see RUNBOOK.md.
+MODEL_NAME = "Qwen/Qwen2.5-VL-7B-Instruct"
+MODEL_REVISION = "cc594898137f460bfe9f0759e9844b3ce807cfb5"
+# The model id clients put in the request body ("model": ...). Matches the
+# example already shipped in openadapt-flow docs/deployment.example.yaml.
+SERVED_MODEL_NAME = "qwen2.5-vl-7b-instruct"
+
+# -- serving pin -------------------------------------------------------------
+# vLLM 0.10.1.1: a version attested to serve Qwen2.5-VL with exactly the flags
+# used below. Bump ONLY together with a verified deploy (RUNBOOK.md).
+VLLM_VERSION = "0.10.1.1"
+
+GPU = "A10G"  # 24 GB. "L4" (24 GB, cheaper, slower) is a drop-in alternative.
+PORT = 8000
+SCALEDOWN_WINDOW_S = 120  # hard cap per ops policy: idle GPU dies within 120 s
+STARTUP_TIMEOUT_S = 20 * 60  # first-ever cold start downloads ~16 GB of weights
+MAX_MODEL_LEN = 16384  # one full-desktop screenshot is ~2.7k vision tokens
+GPU_MEMORY_UTILIZATION = 0.90
+MAX_CONCURRENT_INPUTS = 8
+
+image = (
+    modal.Image.debian_slim(python_version="3.12")
+    .uv_pip_install(
+        f"vllm=={VLLM_VERSION}",
+        "huggingface_hub[hf_transfer]",
+    )
+    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
+)
+
+# Weights cache: survives scale-to-zero, so only the FIRST cold start ever
+# pays the HuggingFace download. Subsequent cold starts load from the volume.
+hf_cache = modal.Volume.from_name(
+    "qwen-grounder-endpoint-hf-cache", create_if_missing=True
+)
+# torch.compile / vLLM artifact cache: shaves repeat cold-start work.
+vllm_cache = modal.Volume.from_name(
+    "qwen-grounder-endpoint-vllm-cache", create_if_missing=True
+)
+
+app = modal.App(APP_NAME)
+
+
+@app.function(
+    image=image,
+    gpu=GPU,
+    scaledown_window=SCALEDOWN_WINDOW_S,
+    timeout=60 * 60,
+    volumes={
+        "/root/.cache/huggingface": hf_cache,
+        "/root/.cache/vllm": vllm_cache,
+    },
+    secrets=[modal.Secret.from_name("qwen-endpoint-token")],
+)
+@modal.concurrent(max_inputs=MAX_CONCURRENT_INPUTS)
+@modal.web_server(port=PORT, startup_timeout=STARTUP_TIMEOUT_S)
+def serve() -> None:
+    """Launch the vLLM OpenAI-compatible server.
+
+    The bearer token is read from the environment injected by the Modal
+    secret ``qwen-endpoint-token`` (key ``TOKEN``). It is passed to vLLM as
+    ``--api-key`` and never printed. A missing secret key fails loudly here
+    rather than starting an unauthenticated server.
+    """
+    import os
+
+    token = os.environ["TOKEN"]  # KeyError => refuse to start without auth
+    if not token.strip():
+        raise RuntimeError(
+            "Secret qwen-endpoint-token has an empty TOKEN; refusing to start "
+            "an unauthenticated server."
+        )
+
+    cmd = [
+        "vllm",
+        "serve",
+        MODEL_NAME,
+        "--revision",
+        MODEL_REVISION,
+        "--served-model-name",
+        SERVED_MODEL_NAME,
+        "--host",
+        "0.0.0.0",
+        "--port",
+        str(PORT),
+        "--api-key",
+        token,
+        "--gpu-memory-utilization",
+        str(GPU_MEMORY_UTILIZATION),
+        "--max-model-len",
+        str(MAX_MODEL_LEN),
+    ]
+    # Popen, not run: web_server expects the function to return once the
+    # port is (eventually) listening; vLLM keeps serving in this process.
+    subprocess.Popen(cmd)
@@ -0,0 +1,30 @@
+# openadapt-flow deployment.yaml snippet: point the grounder at the
+# self-hosted Qwen endpoint on Modal (qwen_endpoint/app.py in openadapt-ops).
+#
+# Merge this `runtime:` block into an operator's deployment config (schema:
+# openadapt_flow/deployment.py; full example: docs/deployment.example.yaml in
+# the openadapt-flow repo). The endpoint URL below is the deterministic Modal
+# web-server URL for workspace `abrichr`; `modal deploy` prints the exact URL.
+#
+# The API key is a REFERENCE: `api_key_env` names an env var, never a literal.
+# Source it from the Keychain at runtime, e.g. via qwen_endpoint/with_token.sh:
+#   ./with_token.sh openadapt-flow run --config deployment.yaml ...
+
+runtime:
+  # Egress opt-in (PHI audit REM-3): the VLM grounder sends the screenshot to
+  # the endpoint. Without this flag the run stays fully local and the grounder
+  # below is refused.
+  allow_model_grounding: true
+
+  # The VLM fallback rung BEHIND the local OCR rung (bring-your-own-model).
+  grounding_model:
+    enabled: true
+    provider: openai_compatible
+    base_url: "https://abrichr--qwen-grounder-endpoint-serve.modal.run/v1"
+    model: "qwen2.5-vl-7b-instruct"
+    api_key_env: "OPENADAPT_FLOW_GROUNDING_API_KEY"
+
+  # PHI mode only (fail-closed): the endpoint host must be allowlisted or the
+  # run stays fully local. Harmless in non-PHI runs.
+  phi_grounding_allowlist:
+    - "abrichr--qwen-grounder-endpoint-serve.modal.run"