From 74da70402724c49c66e2d50f81b7060d5f16d037 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 19:12:02 +0000 Subject: [PATCH 1/2] fix(ci): pin the model server by digest and preflight streamed tool calls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The live-model gate failed on its first real run and blocked the publish. The cause was the pin, not the agent: `server-bNNNN` tags stop at b5350 because the project renamed its tags afterwards, so the newest-looking tag was a year-old build. That build serves plain chat fine and answers every STREAMED tool call with `Cannot use tools with stream` — which is exactly what the executor does, so the run died deep inside the agent with nothing pointing at the runtime. - Pin by digest (b10450). A digest cannot rot into a different meaning the way a naming scheme can. - Add a preflight that sends one streamed tool call and fails with a message naming the image. Verified to discriminate: HTTP 500 on b5350, 200 on the new digest. Re-measured on the new image: the 3B passes; a 1.5B still never calls the tool (0/3 runs, `calls: []`), so the floor stays where it is. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011FKop4At26QqqkwVGEjJur --- .github/workflows/live-model.yml | 53 +++++++++++++++++++++++++++++--- CLAUDE.md | 2 +- 2 files changed, 49 insertions(+), 6 deletions(-) diff --git a/.github/workflows/live-model.yml b/.github/workflows/live-model.yml index 3122004..cb7acd9 100644 --- a/.github/workflows/live-model.yml +++ b/.github/workflows/live-model.yml @@ -23,12 +23,24 @@ on: # Pinned so a rerun behaves the same. Bump deliberately. # -# 3B is the floor, measured rather than guessed: on this exact runtime a -# Qwen2.5-1.5B never called the tool — it answered from imagination, once -# reporting a tool's NAME as the secret — while the 3B called it and returned -# the real value. See the note on the test's phrasing in live-model.test.ts. +# The server is pinned by DIGEST, not by tag, and that is a scar: the +# `server-bNNNN` tags stop at b5350 (May 2025) because the project renamed its +# tags afterwards, so "the newest server-b tag" silently meant a year-old +# build. That build answers every streamed tool call with +# `Cannot use tools with stream` — precisely what this agent does — so the job +# failed and correctly refused to publish. A digest cannot rot into a +# different meaning the way a naming scheme can. The floating tag is +# `ghcr.io/ggml-org/llama.cpp:server`; resolve it and paste the digest here. +# +# Digest below is b10450. +# +# 3B is the floor, measured rather than guessed: on this exact image a +# Qwen2.5-1.5B never called the tool in three runs (`calls: []`) — it answers +# from imagination, once reporting a tool's NAME as the secret — while the 3B +# called it and returned the real value. See the note on the test's phrasing +# in live-model.test.ts. env: - LLAMA_IMAGE: ghcr.io/ggml-org/llama.cpp:server-b5350 + LLAMA_IMAGE: ghcr.io/ggml-org/llama.cpp@sha256:0668d42b64608e32e8d168e7984aabb1557e06e155cfaa268e2b8fa6a55ff841 MODEL_REPO: Qwen/Qwen2.5-3B-Instruct-GGUF MODEL_FILE: qwen2.5-3b-instruct-q4_k_m.gguf @@ -95,6 +107,37 @@ jobs: docker logs llama | tail -50 exit 1 + - name: Check the server can stream a tool call + # The failure this catches already happened once and cost a release: + # an old build served plain chat fine and rejected every streamed tool + # call, so the run died deep inside the agent with nothing pointing at + # the runtime. One request, asserted directly, turns that into a named + # error before the model's judgement is ever involved. + run: | + set -uo pipefail + code=$(curl -s -o /tmp/preflight.json -w '%{http_code}' \ + http://127.0.0.1:8080/v1/chat/completions \ + -H 'content-type: application/json' \ + -d '{ + "model": "preflight", + "stream": true, + "messages": [{"role": "user", "content": "Call the ping tool."}], + "tools": [{ + "type": "function", + "function": { + "name": "ping", + "description": "ping", + "parameters": {"type": "object", "properties": {}} + } + }] + }') + if [ "$code" != "200" ]; then + echo "::error::the server rejected a STREAMED tool call (HTTP $code) — the pinned image is too old for what the agent does; bump LLAMA_IMAGE" + cat /tmp/preflight.json + exit 1 + fi + echo "streamed tool calls are supported" + - name: Run the live-model smoke test # One retry: a small model occasionally wanders, and a single stray run # must not block a publish. A second failure is a real signal. diff --git a/CLAUDE.md b/CLAUDE.md index 8e9b2df..88678de 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,7 +61,7 @@ This repo follows the unified `autoupdate-with-claude` baseline (same template a - `autoupdate.yml` uses `GITHUB_TOKEN` and explicitly dispatches `test.yml` (the `CI` workflow) after PR creation, because events created via `GITHUB_TOKEN` don't trigger `pull_request` workflows. - `autoupdate.yml` dispatches `claude.yml` directly via `workflow_dispatch` instead of relying on an `@claude` PR comment. - Releases stay wired through `release.yml` (npm Trusted Publisher), which fires via `workflow_run` after a successful CI on `main`. There is no `release-on-version-bump.yml` here — it would conflict with the existing tag/publish chain. -- `release.yml` now `needs:` the `live-model.yml` workflow: a smoke test against a real model (llama.cpp on the runner, no key) gates the publish. It also runs on PRs that touch `package.json` / `package-lock.json`, which is where dependency drift arrives. Deliberately NOT nightly — a 3am failure on yesterday's commit gets ignored, and the question it answers matters at publish time. `LLAMA_IMAGE` / `MODEL_*` are pinned so a rerun downloads nothing and behaves the same; bump them on purpose. 3B is the measured floor — a 1.5B answers from imagination instead of calling the tool. +- `release.yml` now `needs:` the `live-model.yml` workflow: a smoke test against a real model (llama.cpp on the runner, no key) gates the publish. It also runs on PRs that touch `package.json` / `package-lock.json`, which is where dependency drift arrives. Deliberately NOT nightly — a 3am failure on yesterday's commit gets ignored, and the question it answers matters at publish time. `LLAMA_IMAGE` / `MODEL_*` are pinned so a rerun downloads nothing and behaves the same; bump them on purpose. `LLAMA_IMAGE` is pinned **by digest**, not by tag: the `server-bNNNN` tags stop at b5350 because the project renamed its tags, so the newest-looking tag was a year-old build that rejects streamed tool calls (`Cannot use tools with stream`) and blocked a release. Resolve `ghcr.io/ggml-org/llama.cpp:server` and paste the digest. A preflight step asserts the server can stream a tool call, so that failure names itself instead of dying inside the agent. 3B is the measured floor — a 1.5B answers from imagination instead of calling the tool (0/3 runs touched it). - All actions pinned to the `@v4` line because the runner image currently lacks `externals/node24`, breaking post-cleanup of `@v5/@v6` actions. Do **not** "fix" any of the above by replacing dispatch calls with comment-based mentions, or by bumping action versions back to `@v5/@v6`. From 65c88ee3684ad07f09c5b7edf3229ad7513767ef Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 19:13:01 +0000 Subject: [PATCH 2/2] ci: run the live-model gate when the gate itself changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Its first version shipped broken and could not be exercised on the PR that introduced it — the paths filter only listed dependency files, so the first real execution was on main, where it blocked a publish. The workflow, the test and its helpers now trigger it too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011FKop4At26QqqkwVGEjJur --- .github/workflows/live-model.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.github/workflows/live-model.yml b/.github/workflows/live-model.yml index cb7acd9..3481c0a 100644 --- a/.github/workflows/live-model.yml +++ b/.github/workflows/live-model.yml @@ -17,9 +17,15 @@ on: branches: [main] # Dependency bumps (including the autoupdate flow's) are the realistic # source of drift; ordinary source PRs are covered by the scripted suite. + # The gate's own definition is in the list because the first version of it + # shipped broken and could not be exercised until it was already on main + # blocking a publish — if you edit the gate, run the gate. paths: - 'package.json' - 'package-lock.json' + - '.github/workflows/live-model.yml' + - 'tests/live-model.test.ts' + - 'tests/helpers/**' # Pinned so a rerun behaves the same. Bump deliberately. #