diff --git a/.github/workflows/live-model.yml b/.github/workflows/live-model.yml index 3122004..3481c0a 100644 --- a/.github/workflows/live-model.yml +++ b/.github/workflows/live-model.yml @@ -17,18 +17,36 @@ on: branches: [main] # Dependency bumps (including the autoupdate flow's) are the realistic # source of drift; ordinary source PRs are covered by the scripted suite. + # The gate's own definition is in the list because the first version of it + # shipped broken and could not be exercised until it was already on main + # blocking a publish — if you edit the gate, run the gate. paths: - 'package.json' - 'package-lock.json' + - '.github/workflows/live-model.yml' + - 'tests/live-model.test.ts' + - 'tests/helpers/**' # Pinned so a rerun behaves the same. Bump deliberately. # -# 3B is the floor, measured rather than guessed: on this exact runtime a -# Qwen2.5-1.5B never called the tool — it answered from imagination, once -# reporting a tool's NAME as the secret — while the 3B called it and returned -# the real value. See the note on the test's phrasing in live-model.test.ts. +# The server is pinned by DIGEST, not by tag, and that is a scar: the +# `server-bNNNN` tags stop at b5350 (May 2025) because the project renamed its +# tags afterwards, so "the newest server-b tag" silently meant a year-old +# build. That build answers every streamed tool call with +# `Cannot use tools with stream` — precisely what this agent does — so the job +# failed and correctly refused to publish. A digest cannot rot into a +# different meaning the way a naming scheme can. The floating tag is +# `ghcr.io/ggml-org/llama.cpp:server`; resolve it and paste the digest here. +# +# Digest below is b10450. +# +# 3B is the floor, measured rather than guessed: on this exact image a +# Qwen2.5-1.5B never called the tool in three runs (`calls: []`) — it answers +# from imagination, once reporting a tool's NAME as the secret — while the 3B +# called it and returned the real value. See the note on the test's phrasing +# in live-model.test.ts. env: - LLAMA_IMAGE: ghcr.io/ggml-org/llama.cpp:server-b5350 + LLAMA_IMAGE: ghcr.io/ggml-org/llama.cpp@sha256:0668d42b64608e32e8d168e7984aabb1557e06e155cfaa268e2b8fa6a55ff841 MODEL_REPO: Qwen/Qwen2.5-3B-Instruct-GGUF MODEL_FILE: qwen2.5-3b-instruct-q4_k_m.gguf @@ -95,6 +113,37 @@ jobs: docker logs llama | tail -50 exit 1 + - name: Check the server can stream a tool call + # The failure this catches already happened once and cost a release: + # an old build served plain chat fine and rejected every streamed tool + # call, so the run died deep inside the agent with nothing pointing at + # the runtime. One request, asserted directly, turns that into a named + # error before the model's judgement is ever involved. + run: | + set -uo pipefail + code=$(curl -s -o /tmp/preflight.json -w '%{http_code}' \ + http://127.0.0.1:8080/v1/chat/completions \ + -H 'content-type: application/json' \ + -d '{ + "model": "preflight", + "stream": true, + "messages": [{"role": "user", "content": "Call the ping tool."}], + "tools": [{ + "type": "function", + "function": { + "name": "ping", + "description": "ping", + "parameters": {"type": "object", "properties": {}} + } + }] + }') + if [ "$code" != "200" ]; then + echo "::error::the server rejected a STREAMED tool call (HTTP $code) — the pinned image is too old for what the agent does; bump LLAMA_IMAGE" + cat /tmp/preflight.json + exit 1 + fi + echo "streamed tool calls are supported" + - name: Run the live-model smoke test # One retry: a small model occasionally wanders, and a single stray run # must not block a publish. A second failure is a real signal. diff --git a/CLAUDE.md b/CLAUDE.md index 8e9b2df..88678de 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,7 +61,7 @@ This repo follows the unified `autoupdate-with-claude` baseline (same template a - `autoupdate.yml` uses `GITHUB_TOKEN` and explicitly dispatches `test.yml` (the `CI` workflow) after PR creation, because events created via `GITHUB_TOKEN` don't trigger `pull_request` workflows. - `autoupdate.yml` dispatches `claude.yml` directly via `workflow_dispatch` instead of relying on an `@claude` PR comment. - Releases stay wired through `release.yml` (npm Trusted Publisher), which fires via `workflow_run` after a successful CI on `main`. There is no `release-on-version-bump.yml` here — it would conflict with the existing tag/publish chain. -- `release.yml` now `needs:` the `live-model.yml` workflow: a smoke test against a real model (llama.cpp on the runner, no key) gates the publish. It also runs on PRs that touch `package.json` / `package-lock.json`, which is where dependency drift arrives. Deliberately NOT nightly — a 3am failure on yesterday's commit gets ignored, and the question it answers matters at publish time. `LLAMA_IMAGE` / `MODEL_*` are pinned so a rerun downloads nothing and behaves the same; bump them on purpose. 3B is the measured floor — a 1.5B answers from imagination instead of calling the tool. +- `release.yml` now `needs:` the `live-model.yml` workflow: a smoke test against a real model (llama.cpp on the runner, no key) gates the publish. It also runs on PRs that touch `package.json` / `package-lock.json`, which is where dependency drift arrives. Deliberately NOT nightly — a 3am failure on yesterday's commit gets ignored, and the question it answers matters at publish time. `LLAMA_IMAGE` / `MODEL_*` are pinned so a rerun downloads nothing and behaves the same; bump them on purpose. `LLAMA_IMAGE` is pinned **by digest**, not by tag: the `server-bNNNN` tags stop at b5350 because the project renamed its tags, so the newest-looking tag was a year-old build that rejects streamed tool calls (`Cannot use tools with stream`) and blocked a release. Resolve `ghcr.io/ggml-org/llama.cpp:server` and paste the digest. A preflight step asserts the server can stream a tool call, so that failure names itself instead of dying inside the agent. 3B is the measured floor — a 1.5B answers from imagination instead of calling the tool (0/3 runs touched it). - All actions pinned to the `@v4` line because the runner image currently lacks `externals/node24`, breaking post-cleanup of `@v5/@v6` actions. Do **not** "fix" any of the above by replacing dispatch calls with comment-based mentions, or by bumping action versions back to `@v5/@v6`.