diff --git a/qwen_endpoint/RESULTS.md b/qwen_endpoint/RESULTS.md new file mode 100644 index 0000000..139724d --- /dev/null +++ b/qwen_endpoint/RESULTS.md @@ -0,0 +1,47 @@ +# Qwen grounder endpoint — verification results + +Status: **NOT DEPLOYED — blocked on the auth secret.** + +Preflight on 2026-08-12 found Modal auth present (`~/.modal.toml`, profile +`abrichr`) but no Modal secret named `qwen-endpoint-token`. Per policy the +agent does not create or handle the token; the founder command is in +`RUNBOOK.md` section 0 (and in `NEEDS_YOU.md`). Once the secret exists, run +RUNBOOK sections 1–2 and fill this file in. + +## Endpoint + +| What | Value | +|---|---| +| Deployed | no (pending secret) | +| Model | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5`, bf16 (no quantization) | +| GPU | A10G 24 GB | +| Cold-start latency | _(measure: RUNBOOK 2.2)_ | +| Warm request latency | _(measure: RUNBOOK 2.3)_ | +| Unauthed request | _(expect 401: RUNBOOK 2.1)_ | +| Scaled to zero after idle | _(expect yes: RUNBOOK 2.4)_ | +| Verification GPU spend | _(cap $5.00; expected ~$0.20–0.30)_ | + +## Grounder smoke (5 requests, 2 committed flow fixtures) + +Produced by `smoke_grounder.py` through openadapt-flow's real +`OpenAICompatibleGrounder`. Fixtures: `benchmark/dense_surface/ +record_seed1.png` (2240x3702) and `benchmark/dense_surface/ +replay_native_arial_seed1.png` (1120x1858) from the openadapt-flow repo. + +| fixture | intent | verdict | point | latency_s | est_cost_usd | +|---|---|---|---|---|---| +| _(pending deploy)_ | | | | | | + +Summary: _(hit / miss / abstain counts, median latency, cost per request)_ + +Interpretation notes, fixed in advance: + +* This is a smoke (wire works end-to-end), not an accuracy claim. The full + accuracy probe runs against Together in a sibling effort — compare cost + and hit rate there before choosing a default grounding backend. +* Qwen VL models answer in the coordinate frame of the server-side + (possibly resized) image; a systematic offset on the hi-dpi fixture is + recorded as `miss`, not tuned away. +* `abstain` (grounder returns None) is the fail-safe path working: any + transport error, non-200, or `{"x": null}` reply must halt the ladder, + never click. diff --git a/qwen_endpoint/RUNBOOK.md b/qwen_endpoint/RUNBOOK.md new file mode 100644 index 0000000..d16981b --- /dev/null +++ b/qwen_endpoint/RUNBOOK.md @@ -0,0 +1,136 @@ +# Qwen grounder endpoint — deploy / verify / teardown runbook + +Self-hosted Qwen VL endpoint on Modal for openadapt-flow's +`OpenAICompatibleGrounder`. OpenAI-compatible, bearer-token auth, +scale-to-zero (idle GPU dies within 120 s, no warm pool). + +| What | Value | +|---|---| +| App | `qwen-grounder-endpoint` (`qwen_endpoint/app.py`) | +| Model | `Qwen/Qwen2.5-VL-7B-Instruct` @ `cc594898137f460bfe9f0759e9844b3ce807cfb5` | +| Served model id | `qwen2.5-vl-7b-instruct` | +| Serving | vLLM `0.10.1.1`, bf16, `--max-model-len 16384` | +| GPU | one A10G (24 GB) | +| URL | `https://--qwen-grounder-endpoint-serve.modal.run/v1` (printed by `modal deploy`; workspace `abrichr`) | +| Auth | Modal secret `qwen-endpoint-token`, key `TOKEN`; unauthed requests get HTTP 401 from vLLM `--api-key` | +| Scale-to-zero | `scaledown_window=120`, no `min_containers` (guarded by `tests/test_qwen_endpoint.py`) | + +## 0. One-time: create the auth secret (FOUNDER ONLY) + +Agents never create, read, or copy this token. The founder runs, once: + +```bash +python3 -c "import secrets;print(secrets.token_urlsafe(32))" | { read t; modal secret create qwen-endpoint-token TOKEN="$t"; security add-generic-password -s oa-qwen-endpoint-token -a modal -w "$t"; } +``` + +This mints the token, stores it as the Modal secret `qwen-endpoint-token` +(read only by the serving container) and as the macOS Keychain item +`oa-qwen-endpoint-token` (read only by `with_token.sh` at client runtime). +The value itself is never displayed. + +Everything below assumes `modal secret list` shows `qwen-endpoint-token`. + +## 1. Deploy + +```bash +cd /Users/abrichr/oa/src/openadapt-ops +modal deploy qwen_endpoint/app.py +``` + +`modal deploy` prints the web URL. The FIRST-ever cold start additionally +downloads ~16 GB of weights into the `qwen-grounder-endpoint-hf-cache` +volume (one-time, a few minutes); later cold starts load from the volume. + +## 2. Verify + +All authed calls go through the launcher so the token stays in the Keychain: + +```bash +cd /Users/abrichr/oa/src/openadapt-ops/qwen_endpoint +BASE=https://abrichr--qwen-grounder-endpoint-serve.modal.run/v1 +``` + +1. **Unauthed => 401** (no launcher, no token): + + ```bash + curl -s -o /dev/null -w '%{http_code}\n' "$BASE/models" # expect 401 + curl -s -o /dev/null -w '%{http_code}\n' -X POST "$BASE/chat/completions" \ + -H 'Content-Type: application/json' \ + -d '{"model":"qwen2.5-vl-7b-instruct","messages":[{"role":"user","content":"hi"}]}' # expect 401 + ``` + +2. **Cold-start latency** (first authed call after idle; time it): + + ```bash + time ./with_token.sh sh -c 'curl -s -H "Authorization: Bearer $OPENADAPT_FLOW_GROUNDING_API_KEY" "$0/models"' "$BASE" + ``` + +3. **Warm latency + well-formed vision output** — repeat the same call, then + run the grounder smoke (5 image requests through the real flow client): + + ```bash + ./with_token.sh uv run --project /Users/abrichr/oa/src/openadapt-flow \ + python3 smoke_grounder.py --base-url "$BASE" \ + --flow-repo /Users/abrichr/oa/src/openadapt-flow --gpu-usd-per-hour 1.10 + ``` + + Paste the printed table into `RESULTS.md`. + +4. **Scale-to-zero**: wait >2 minutes with no traffic, then: + + ```bash + modal app list # qwen-grounder-endpoint shows 0 running containers + modal container list # no container for the app + ``` + +## 3. Teardown + +```bash +modal app stop qwen-grounder-endpoint # removes the deployment + URL +# optional, reclaims ~16 GB of weight-cache storage: +modal volume delete qwen-grounder-endpoint-hf-cache +modal volume delete qwen-grounder-endpoint-vllm-cache +``` + +Founder-only, if retiring the endpoint for good: `modal secret delete +qwen-endpoint-token` and `security delete-generic-password -s +oa-qwen-endpoint-token -a modal`. + +## 4. Cost math + +Rates are the Modal list prices as of 2026-08 — re-check +`https://modal.com/pricing` before trusting a forecast. + +| Item | Math | Cost | +|---|---|---| +| A10G GPU-second | ~$1.10/h | ~$0.000306/s | +| Cold start (volume-cached weights) | ~120–240 s | $0.04–0.08 | +| First-ever cold start (HF download) | +~180 s | +~$0.06 | +| Warm grounding request | ~2–6 s | <$0.002 | +| Idle tail after a burst | exactly 120 s | $0.037 | +| **Idle deployment (steady state)** | 0 GPU-s | **$0.00 GPU** + volume storage (~16 GB, order of $0.50/mo; confirm in dashboard) | +| Full verification pass (steps 1–4 above) | ~10–15 min GPU | **~$0.20–0.30** (cap: $5.00) | + +Example month — 500 grounding calls in 50 bursts: 50 × (cold 180 s + calls +~30 s + idle 120 s) ≈ 4.6 GPU-hours ≈ **$5/mo**. The same calls against a +hosted per-token API are the sibling Together effort's numbers; compare +before committing either way. + +## 5. Pinned-version upgrade path (do NOT bump casually) + +* Any bump of `MODEL_REVISION`, `VLLM_VERSION`, or the model itself requires + a full section-2 verification pass in the same PR, with measured numbers. +* Designated next model: `Qwen/Qwen3-VL-8B-Instruct` @ + `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b` — needs `vllm>=0.11.0`, and at + ~17 GB bf16 it is tight on 24 GB: expect to lower `--max-model-len`, or + use the official FP8 checkpoint on an L4/Ada GPU (A10G/Ampere has no + native FP8). +* `tests/test_qwen_endpoint.py` pins the safety invariants (auth secret + name, 120 s scaledown, no warm pool, pinned revision); it must keep + passing untouched. + +## 6. Client wiring + +`deployment.snippet.yaml` in this directory is the operator-facing flow +config. Run flow under the launcher so the env var named by `api_key_env` +exists: `./with_token.sh openadapt-flow run --config deployment.yaml ...`. diff --git a/qwen_endpoint/app.py b/qwen_endpoint/app.py new file mode 100644 index 0000000..1cbd1e2 --- /dev/null +++ b/qwen_endpoint/app.py @@ -0,0 +1,131 @@ +"""Modal app: Qwen VL grounding endpoint (OpenAI-compatible, scale-to-zero). + +Serves ``Qwen/Qwen2.5-VL-7B-Instruct`` (pinned revision) via vLLM behind an +OpenAI-compatible ``/v1/chat/completions`` route, sized for one 24 GB GPU. + +Design constraints (enforced by ``tests/test_qwen_endpoint.py``): + +* Scale-to-zero: ``scaledown_window`` is at most 120 s; no container floor, + no warm pool. An idle deployment costs $0 in GPU time. +* Auth: every request must carry ``Authorization: Bearer `` where the + token comes from the Modal secret ``qwen-endpoint-token`` (key ``TOKEN``). + vLLM's ``--api-key`` flag rejects anything else with HTTP 401. The token is + read INSIDE the container from the secret; it never appears in this repo, + in logs, or in the deploy output. +* Pinned model: the exact HuggingFace revision is pinned below so a re-deploy + serves byte-identical weights. + +Deploy / verify / teardown: see ``qwen_endpoint/RUNBOOK.md``. + +Client wiring: openadapt-flow's ``OpenAICompatibleGrounder`` points at +``https://--qwen-grounder-endpoint-serve.modal.run/v1`` — see +``qwen_endpoint/deployment.snippet.yaml``. +""" + +import subprocess + +import modal + +APP_NAME = "qwen-grounder-endpoint" + +# -- model pin --------------------------------------------------------------- +# Qwen2.5-VL-7B-Instruct: the largest Qwen VL known to serve reliably on one +# 24 GB GPU under vLLM in bf16 (~15.5 GB weights + KV cache at 16k context). +# Revision = HF main as read on 2026-08-12 (last modified 2025-04-06). +# +# Qwen3-VL-8B-Instruct (revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b) is +# the designated upgrade once a deploy can be GPU-verified: it needs +# vllm>=0.11.0 and is tighter on 24 GB (~17 GB bf16 weights); see RUNBOOK.md. +MODEL_NAME = "Qwen/Qwen2.5-VL-7B-Instruct" +MODEL_REVISION = "cc594898137f460bfe9f0759e9844b3ce807cfb5" +# The model id clients put in the request body ("model": ...). Matches the +# example already shipped in openadapt-flow docs/deployment.example.yaml. +SERVED_MODEL_NAME = "qwen2.5-vl-7b-instruct" + +# -- serving pin ------------------------------------------------------------- +# vLLM 0.10.1.1: a version attested to serve Qwen2.5-VL with exactly the flags +# used below. Bump ONLY together with a verified deploy (RUNBOOK.md). +VLLM_VERSION = "0.10.1.1" + +GPU = "A10G" # 24 GB. "L4" (24 GB, cheaper, slower) is a drop-in alternative. +PORT = 8000 +SCALEDOWN_WINDOW_S = 120 # hard cap per ops policy: idle GPU dies within 120 s +STARTUP_TIMEOUT_S = 20 * 60 # first-ever cold start downloads ~16 GB of weights +MAX_MODEL_LEN = 16384 # one full-desktop screenshot is ~2.7k vision tokens +GPU_MEMORY_UTILIZATION = 0.90 +MAX_CONCURRENT_INPUTS = 8 + +image = ( + modal.Image.debian_slim(python_version="3.12") + .uv_pip_install( + f"vllm=={VLLM_VERSION}", + "huggingface_hub[hf_transfer]", + ) + .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"}) +) + +# Weights cache: survives scale-to-zero, so only the FIRST cold start ever +# pays the HuggingFace download. Subsequent cold starts load from the volume. +hf_cache = modal.Volume.from_name( + "qwen-grounder-endpoint-hf-cache", create_if_missing=True +) +# torch.compile / vLLM artifact cache: shaves repeat cold-start work. +vllm_cache = modal.Volume.from_name( + "qwen-grounder-endpoint-vllm-cache", create_if_missing=True +) + +app = modal.App(APP_NAME) + + +@app.function( + image=image, + gpu=GPU, + scaledown_window=SCALEDOWN_WINDOW_S, + timeout=60 * 60, + volumes={ + "/root/.cache/huggingface": hf_cache, + "/root/.cache/vllm": vllm_cache, + }, + secrets=[modal.Secret.from_name("qwen-endpoint-token")], +) +@modal.concurrent(max_inputs=MAX_CONCURRENT_INPUTS) +@modal.web_server(port=PORT, startup_timeout=STARTUP_TIMEOUT_S) +def serve() -> None: + """Launch the vLLM OpenAI-compatible server. + + The bearer token is read from the environment injected by the Modal + secret ``qwen-endpoint-token`` (key ``TOKEN``). It is passed to vLLM as + ``--api-key`` and never printed. A missing secret key fails loudly here + rather than starting an unauthenticated server. + """ + import os + + token = os.environ["TOKEN"] # KeyError => refuse to start without auth + if not token.strip(): + raise RuntimeError( + "Secret qwen-endpoint-token has an empty TOKEN; refusing to start " + "an unauthenticated server." + ) + + cmd = [ + "vllm", + "serve", + MODEL_NAME, + "--revision", + MODEL_REVISION, + "--served-model-name", + SERVED_MODEL_NAME, + "--host", + "0.0.0.0", + "--port", + str(PORT), + "--api-key", + token, + "--gpu-memory-utilization", + str(GPU_MEMORY_UTILIZATION), + "--max-model-len", + str(MAX_MODEL_LEN), + ] + # Popen, not run: web_server expects the function to return once the + # port is (eventually) listening; vLLM keeps serving in this process. + subprocess.Popen(cmd) diff --git a/qwen_endpoint/deployment.snippet.yaml b/qwen_endpoint/deployment.snippet.yaml new file mode 100644 index 0000000..7329c5d --- /dev/null +++ b/qwen_endpoint/deployment.snippet.yaml @@ -0,0 +1,30 @@ +# openadapt-flow deployment.yaml snippet: point the grounder at the +# self-hosted Qwen endpoint on Modal (qwen_endpoint/app.py in openadapt-ops). +# +# Merge this `runtime:` block into an operator's deployment config (schema: +# openadapt_flow/deployment.py; full example: docs/deployment.example.yaml in +# the openadapt-flow repo). The endpoint URL below is the deterministic Modal +# web-server URL for workspace `abrichr`; `modal deploy` prints the exact URL. +# +# The API key is a REFERENCE: `api_key_env` names an env var, never a literal. +# Source it from the Keychain at runtime, e.g. via qwen_endpoint/with_token.sh: +# ./with_token.sh openadapt-flow run --config deployment.yaml ... + +runtime: + # Egress opt-in (PHI audit REM-3): the VLM grounder sends the screenshot to + # the endpoint. Without this flag the run stays fully local and the grounder + # below is refused. + allow_model_grounding: true + + # The VLM fallback rung BEHIND the local OCR rung (bring-your-own-model). + grounding_model: + enabled: true + provider: openai_compatible + base_url: "https://abrichr--qwen-grounder-endpoint-serve.modal.run/v1" + model: "qwen2.5-vl-7b-instruct" + api_key_env: "OPENADAPT_FLOW_GROUNDING_API_KEY" + + # PHI mode only (fail-closed): the endpoint host must be allowlisted or the + # run stays fully local. Harmless in non-PHI runs. + phi_grounding_allowlist: + - "abrichr--qwen-grounder-endpoint-serve.modal.run" diff --git a/qwen_endpoint/smoke_grounder.py b/qwen_endpoint/smoke_grounder.py new file mode 100755 index 0000000..9086e3e --- /dev/null +++ b/qwen_endpoint/smoke_grounder.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Smoke-prove openadapt-flow's OpenAICompatibleGrounder against the endpoint. + +Runs 5 grounding requests through ``openadapt_flow.runtime.grounder. +OpenAICompatibleGrounder`` (the real client class, not a re-implementation) +against 2 screenshot fixtures COMMITTED in the openadapt-flow repo: + +* ``benchmark/dense_surface/record_seed1.png`` (2240x3702, hi-dpi) +* ``benchmark/dense_surface/replay_native_arial_seed1.png`` (1120x1858, 1x) + +Both show the MockMed patient-records list (demo, all data fake). Each case +asks for the "Open" button on one named patient's row and judges the proposal +against a hand-verified expected rectangle for that row's Open button: + +* abstain -> the grounder returned None (fail-safe path exercised) +* hit -> proposed point inside the expected rectangle +* miss -> proposed point outside it + +This is a SMOKE test (does the wire work end-to-end), not the accuracy probe. +The full accuracy probe runs against Together in a sibling effort. Note: +Qwen VL models answer in the coordinate frame of the (possibly resized) +image the server preprocessed, so a systematic offset on the hi-dpi fixture +shows up here as `miss` — record it, do not tune around it. + +Usage (token comes from the Keychain via the launcher; never passed on argv): + + ./with_token.sh python3 smoke_grounder.py \ + --base-url https://--qwen-grounder-endpoint-serve.modal.run/v1 \ + --model qwen2.5-vl-7b-instruct \ + --flow-repo /Users/abrichr/oa/src/openadapt-flow \ + --gpu-usd-per-hour 1.10 + +Requires ``openadapt_flow`` importable (run under the flow repo's venv, e.g. +``uv run --project ...``) plus ``httpx`` (a flow dependency). +""" + +from __future__ import annotations + +import argparse +import os +import statistics +import sys +import time +from pathlib import Path + +# Each case: (fixture relative path, intent, ocr_text, expected rect x0,y0,x1,y1) +# Rectangles are in ORIGINAL fixture pixels, hand-verified 2026-08-12 against +# the committed PNGs; they cover the Open button on the named patient's row +# with a few pixels of margin. +CASES = [ + ( + "benchmark/dense_surface/record_seed1.png", + "Click Open in the row for patient Halloran, Karen (MRN MG584224)", + "Open", + (1975, 195, 2140, 260), + ), + ( + "benchmark/dense_surface/record_seed1.png", + "Click Open in the row for patient Delgado, Edward (MRN MG901312)", + "Open", + (1975, 1415, 2140, 1485), + ), + ( + "benchmark/dense_surface/record_seed1.png", + "Click Open in the row for patient Kowalski, Maria (MRN RC571054)", + "Open", + (1975, 3245, 2140, 3320), + ), + ( + "benchmark/dense_surface/replay_native_arial_seed1.png", + "Click Open in the row for patient Ferreira, Susan (MRN PT994939)", + "Open", + (1000, 170, 1100, 215), + ), + ( + "benchmark/dense_surface/replay_native_arial_seed1.png", + "Click Open in the row for patient Whitfield, Philip (MRN PT560165)", + "Open", + (1000, 1800, 1100, 1850), + ), +] + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--base-url", required=True, help=".../v1 root of the endpoint") + parser.add_argument("--model", default="qwen2.5-vl-7b-instruct") + parser.add_argument("--flow-repo", required=True, help="openadapt-flow checkout") + parser.add_argument("--timeout", type=float, default=120.0) + parser.add_argument( + "--gpu-usd-per-hour", + type=float, + default=1.10, + help="GPU price used for the per-request cost column (A10G default)", + ) + args = parser.parse_args() + + api_key = os.environ.get("OPENADAPT_FLOW_GROUNDING_API_KEY", "") + if not api_key: + print( + "OPENADAPT_FLOW_GROUNDING_API_KEY is not set. Run via ./with_token.sh " + "so the token is sourced from the Keychain at runtime.", + file=sys.stderr, + ) + return 2 + + try: + from openadapt_flow.runtime.grounder import OpenAICompatibleGrounder + except ImportError as exc: + print( + f"Cannot import openadapt_flow ({exc}). Run under the flow repo's " + "environment, e.g.: ./with_token.sh uv run --project " + f"{args.flow_repo} python3 {Path(__file__).name} ...", + file=sys.stderr, + ) + return 2 + + flow_repo = Path(args.flow_repo) + grounder = OpenAICompatibleGrounder( + base_url=args.base_url, + model=args.model, + api_key=api_key, + timeout=args.timeout, + ) + + rows = [] + latencies = [] + for fixture, intent, ocr_text, rect in CASES: + png_path = flow_repo / fixture + screen_png = png_path.read_bytes() + t0 = time.monotonic() + match = grounder.locate(screen_png, intent, ocr_text=ocr_text) + dt = time.monotonic() - t0 + latencies.append(dt) + if match is None: + verdict, point = "abstain", "-" + else: + x, y = match.point + x0, y0, x1, y1 = rect + verdict = "hit" if (x0 <= x <= x1 and y0 <= y <= y1) else "miss" + point = f"({x}, {y})" + rows.append((fixture.rsplit("/", 1)[-1], intent, verdict, point, dt)) + print(f"[{verdict:7s}] {dt:6.2f}s {point:16s} {intent}") + + # GPU-time cost attribution: wall latency plus the amortized share of the + # 120 s post-burst idle window, at the given hourly rate. + usd_per_s = args.gpu_usd_per_hour / 3600.0 + idle_share_s = 120.0 / len(CASES) + print("\n| fixture | intent | verdict | point | latency_s | est_cost_usd |") + print("|---|---|---|---|---|---|") + for name, intent, verdict, point, dt in rows: + cost = (dt + idle_share_s) * usd_per_s + print(f"| {name} | {intent} | {verdict} | {point} | {dt:.2f} | {cost:.4f} |") + + n = len(rows) + hits = sum(1 for r in rows if r[2] == "hit") + misses = sum(1 for r in rows if r[2] == "miss") + abstains = sum(1 for r in rows if r[2] == "abstain") + total_cost = (sum(latencies) + 120.0) * usd_per_s + print( + f"\n{n} requests: {hits} hit / {misses} miss / {abstains} abstain; " + f"latency median {statistics.median(latencies):.2f}s " + f"(min {min(latencies):.2f}s, max {max(latencies):.2f}s); " + f"burst GPU cost incl. one 120s idle window ~${total_cost:.4f} " + f"(${total_cost / n:.4f}/request)" + ) + # Smoke PASSES when the wire works: every request either grounded or + # cleanly abstained, and at least one call actually grounded. + return 0 if (hits + misses + abstains == n and hits > 0) else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/qwen_endpoint/with_token.sh b/qwen_endpoint/with_token.sh new file mode 100755 index 0000000..a74e070 --- /dev/null +++ b/qwen_endpoint/with_token.sh @@ -0,0 +1,17 @@ +#!/bin/sh +# Run a command with OPENADAPT_FLOW_GROUNDING_API_KEY sourced from the macOS +# Keychain item `oa-qwen-endpoint-token` (account `modal`) at runtime. +# +# The token value never touches the shell history, argv of the child, logs, +# or any file: it is exported as an environment variable only, and this +# script never echoes it. +# +# Usage: +# ./with_token.sh curl ... # authed curl +# ./with_token.sh python3 smoke_grounder.py ... # authed grounder smoke +set -eu + +OPENADAPT_FLOW_GROUNDING_API_KEY="$(security find-generic-password -s oa-qwen-endpoint-token -a modal -w)" +export OPENADAPT_FLOW_GROUNDING_API_KEY + +exec "$@" diff --git a/tests/test_qwen_endpoint.py b/tests/test_qwen_endpoint.py new file mode 100644 index 0000000..8ba9c5e --- /dev/null +++ b/tests/test_qwen_endpoint.py @@ -0,0 +1,74 @@ +"""Pin the safety invariants of the Qwen grounder endpoint (qwen_endpoint/). + +The Modal app must stay scale-to-zero, bearer-token-authenticated, and +version-pinned. These tests read the app source with ``ast`` (the ``modal`` +package is deliberately NOT a docs-CI dependency), so a PR that weakens an +invariant fails here rather than silently deploying an expensive or +unauthenticated endpoint. +""" + +import ast +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +APP_PATH = ROOT / "qwen_endpoint" / "app.py" +APP_SOURCE = APP_PATH.read_text() +APP_TREE = ast.parse(APP_SOURCE) + + +def _constant(name): + """Return the value of a module-level ``NAME = `` assignment.""" + for node in APP_TREE.body: + if isinstance(node, ast.Assign): + for target in node.targets: + if isinstance(target, ast.Name) and target.id == name: + return ast.literal_eval(node.value) + raise AssertionError(f"{name} is not a module-level literal in app.py") + + +def test_scale_to_zero_idle_window_at_most_120s(): + assert _constant("SCALEDOWN_WINDOW_S") <= 120 + + +def test_no_warm_pool_or_minimum_containers(): + # Any of these would keep a GPU container (and its bill) alive while idle. + for forbidden in ("min_containers", "keep_warm", "buffer_containers"): + assert forbidden not in APP_SOURCE, f"{forbidden} defeats scale-to-zero" + + +def test_auth_comes_from_the_named_modal_secret(): + assert 'modal.Secret.from_name("qwen-endpoint-token")' in APP_SOURCE + # The serving code must pass the token to vLLM's built-in auth gate. + assert '"--api-key"' in APP_SOURCE + assert 'os.environ["TOKEN"]' in APP_SOURCE + + +def test_no_token_literal_in_source(): + # The token is 32 url-safe random bytes (~43 chars). Nothing resembling + # a long secret literal may appear in the app source. + for match in re.findall(r'"([A-Za-z0-9_\-]{40,})"', APP_SOURCE): + if re.fullmatch(r"[0-9a-f]{40}", match): + continue # a git/HF commit pin, not a secret + raise AssertionError(f"suspicious long literal in app.py: {match[:8]}...") + + +def test_model_revision_is_a_full_commit_pin(): + assert re.fullmatch(r"[0-9a-f]{40}", _constant("MODEL_REVISION")) + assert _constant("MODEL_NAME") == "Qwen/Qwen2.5-VL-7B-Instruct" + + +def test_vllm_version_is_exact_pinned(): + assert re.fullmatch(r"[0-9]+(\.[0-9]+)+", _constant("VLLM_VERSION")) + assert 'f"vllm=={VLLM_VERSION}"' in APP_SOURCE + + +def test_gpu_is_a_single_24gb_class_card(): + assert _constant("GPU") in {"A10G", "L4"} + + +def test_flow_snippet_names_the_served_model_and_env_reference(): + snippet = (ROOT / "qwen_endpoint" / "deployment.snippet.yaml").read_text() + assert f'model: "{_constant("SERVED_MODEL_NAME")}"' in snippet + assert 'api_key_env: "OPENADAPT_FLOW_GROUNDING_API_KEY"' in snippet + assert "allow_model_grounding: true" in snippet