A missing cmp or python3 said the engine was broken, not the tool absent

[mfethe1]

tests/run.sh:10 states the contract:

never treats a missing prerequisite as a pass — it says SKIP, loudly.

and the curl guard says it again, in more words, for exactly this reason:

The worker the downloader generates is curl, so without curl these three report the downloader broken when what is missing is a tool — which is the one thing this suite says it must never do.

cmp and the python3 behind range_server.py were not held to that. This brings them into line.

Evidence

Captured on Windows 11 / Ryzen 7 3700X (Zen 2) / MSYS2 UCRT64 / gcc 16.2.0. The before side is the merge-base 7f1fbba in a sibling worktree, the after side is this branch; each tool was hidden by moving its binary aside for the duration of both runs, so the two sides differ only in the commit. LC_ALL=C, TERM and COLUMNS pinned, identical grep filter applied to both.

cmp (diffutils) not installed — before, eight checks report FAIL and name the engine. expert cache changes results is the report of a bit-identity violation in the expert cache, and it fired because /usr/bin/cmp was absent. After, the same eight report SKIP and name the missing tool. The pass count is unchanged at 36; only the eight false failures move.

Before	After

python3 not installed — before, four checks report range server did not start, blaming the download script for a missing interpreter. Note that convert.py and serve, which are guarded, correctly SKIP in the very same frame: the inconsistency is visible without leaving the image. After, all five download-script checks SKIP and name the tool, consistent with the checks beside them.

Before	After

Why this matters more than a cosmetic label

Every one of those FAIL lines names the engine or the download script. expert cache changes results is exactly what this suite is for — and in a log it is indistinguishable from the real defect. mla_use_nope: false was read as NoPE and skipped the rotation names the precise bug #27 fixed.

A first-time Windows contributor's first run of this suite currently accuses the expert cache of returning different logits, on a clean checkout, because diffutils is not installed.

UCRT64 is the documented Windows build path (#36) and ships neither diffutils nor python3, so this is the default first experience there, not an exotic configuration.

The change

cmp is guarded once and reached through same(), which returns 0 / 1 / 2 for identical / differ / no-tool. Call sites that could only say PASS or FAIL now have somewhere to put the third case. cmp's own exit 2 ("could not read a file") folds into 1 — which is what if cmp -s already did with it — so no existing verdict moves.

The download section already had the right shape: NO_CURL skipped all five of its checks. That flag becomes DL_MISS and carries the reason rather than a bare 1; python3 joins curl in setting it, and the fixture python3 builds is guarded behind it too.

The SIMD check is deliberately left alone. With no cmp it falls through to the fp-noise bound, which is a real verdict on its own and needs no cmp. It loses the bit-identity claim and nothing else — so it keeps running rather than skipping.

Full results

	before	after
cmp hidden	36 passed, 8 failed, 13 skipped	36 passed, 0 failed, 21 skipped
python3 hidden	7 passed, 4 failed, 19 skipped	7 passed, 0 failed, 24 skipped
both present	44 passed, 0 failed, 13 skipped	unchanged, exit 0

bash -n clean. No file outside tests/run.sh is touched.

Two things noted and deliberately not included

Separate concerns; happy to send either as its own PR.

1. .gitignore has no *.exe or *.dll. It covers waste, libwaste.a, *.so, *.dylib, sweep — so a native Windows build leaves waste.exe, sweep.exe, libwaste.dll and libwastevq.dll untracked.
2. A few python3 calls inside the engine and rotary blocks are still unguarded (the fp-noise comparators, make_test_container.py). Same class of bug — but I did not observe them failing, because by the time this machine had a container-capable run python3 was installed. This project measures rather than asserts, so I have left out a failure I did not actually see.

Context

Found while standing up a native Windows x86 measurement box (128 GB, Zen 2, PCIe 4.0 NVMe) to work on Gate 7 and the AVX2 side of #32/#38. This was the first thing in the way. Related to #36, which is where the "gaps no CI job can reach" framing comes from.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B5eVMiauR4Lkt8Dhc67MNb

[marcobambini]

Merged. Reproduced here first, because the whole point of this change is the
behaviour when a tool is absent, which no CI job exercises — so a green run
says nothing about it either way.

Both tools hidden by a PATH holding symlinks to everything else, so the two
sides differ only in the commit. This machine has real Kimi-Linear and K3
containers, so the counts are larger than yours:

cmp hidden

	main	this PR
	42 passed, 8 failed, 8 skipped	42 passed, 0 failed, 16 skipped

The eight on main, verbatim: resume, no-range fallback,
expert cache changes results, read-ahead changes results,
router lookahead changes results, purgeable slots change results,
mla_use_nope: false was read as NoPE and skipped the rotation,
rope_scaling null or {} did not load as plain RoPE. Every one becomes a SKIP
naming diffutils, and the 42 passes do not move.

python3 hidden — the four range server did not start failures all become
SKIPs naming the interpreter.

Your reading of why this matters more than a label is right and is why it went
in as it stands: expert cache changes results is the report of a bit-identity
violation in the expert cache, and in a log it is indistinguishable from the
real defect. mla_use_nope names the precise bug #27 fixed.

The part you could not observe, observed

You wrote:

A few python3 calls inside the engine and rotary blocks are still unguarded
... I did not observe them failing, because by the time this machine had a
container-capable run python3 was installed.

Leaving out a failure you had not seen was the right call. It also means the
second half was invisible from your machine and visible from this one, which
has both a container and a way to hide the interpreter. With python3 hidden,
this PR still ends at 13 passed, 11 failed, 24 skipped:

FAIL  chunked prefill diverges
FAIL  quantized storage changes results
FAIL  SIMD backend diverges from the CPU baseline
FAIL  engine diverges from the oracle
FAIL  default budget off the rule (no output)
FAIL  format_version not enforced (rc=1 rc2=1)
FAIL  default budget not capped on K3 (no output)
FAIL  info describes something else (no output)
FAIL  params_total off the rule (no output)
FAIL  info describes something else on K3 (no output)
FAIL  params_total off the rule on K3 (no output)

Same defect, different block: the fp-noise comparators and the waste info /
waste plan parsers are python3, and without it they accuse the engine of
diverging from its own oracle. (no output) is the tell — the check never ran.

That is not a criticism of this PR, which does what it says and does it
cleanly. It is the rest of the same contract, and it is now measured rather
than suspected, so it can be closed on purpose. I would rather have it as its
own change than fold it into this one after the fact. Tracking it in #36 with
the reproduction, unless you would like to send it — you found the class.

On the two you set aside

.gitignore has no *.exe or *.dll: correct, and worth a separate one-liner.

53 passed / 0 failed / 6 skipped here with everything installed, unchanged from
the merge-base. Thank you — this is the report I should have acted on faster,
and the one where being left at action_required cost the most.