perf(healthcheck): probe every port concurrently, and report through exit status

[Marinski]

The problem

healthcheck.sh probed the ports one after another, so the check's wall clock
was the product of the probe timeout and the terminal count.
PROBE_TIMEOUT_SECONDS is 3, and our bulk VM carries 24 terminals: 72s against
the compose timeout: 30s.

Docker kills a check that overruns and records

Health check exceeded timeout (30s)

instead of the verdict. That output is worse than useless, because a supervisor
cannot tell it apart from a dead VM. It is also most likely exactly when it
hurts most: while a VM is booting, every port is slow, so a VM that was merely
starting looked identical to one that had crashed, and the watchdog recreated
VMs that were doing nothing wrong.

The change

One background job per port. The worst case is now one probe timeout regardless
of how many terminals the VM carries.

• Each job writes its verdict to its own file under a per-run directory, and the
  parent aggregates after wait. No shared state, so nothing to race on.
• A job that dies before writing its verdict is read as down, matching the
  status whitelist already in the file: an unexplained probe is never a silent
  pass.
• The slow/hung counters from this PR's base keep working, because each port's
  counter file is that port's alone. The one thing a background job cannot do is
  set a variable the parent will see, so a job that cannot write its counter
  leaves a marker file and the parent still prints
  [slow-state unwritable: hung detection off].

The worst case is one port's host walk, not one probe: each job still
tries the leased VM IP and then the two fallbacks in turn, so the bound is
PROBE_TIMEOUT_SECONDS × 3 = 9s. Measured at exactly 9s. Raising the timeout
to 10 would put it back at 30s, so the code and the docs both say so.

Measured under busybox ash and under the container's own dash:

ports	probe	sequential would be	this
24	3s	72s	one host walk
100	2s	200s	4s

Why the verdicts are exit statuses and not files

This is the part worth your attention, because I had it wrong first.

The obvious implementation gives each job a verdict file under a mktemp -d.
Counter-review reproduced what that does on a bad day. Six ports, all
answering HTTP 200, with /tmp full:

scratch-file version:  DOWN ports: 6600 6601 6602 6603 6604 6605   rc=1
sequential baseline:   ok all ports up: 6600 ... 6605              rc=0

The write fails, the parent finds no verdict, and its fail-closed rule turns
"I cannot write my own scratch file" into "every terminal on this VM is down".
Ten of those in a row and the watchdog recreates a healthy VM, destroying two
dozen running backtests, because /tmp filled up. Our / is at 91%. The
sequential loop it replaces had no disk dependency and neither may this.

So each job returns 0 up, 1 busy, 2 hung, 3 dead, 4 busy-but-the-
counter-could-not-be-written, and the parent reads them back with wait in
port order. Anything else, such as a job killed by a signal, is unknown and
fails closed as down. No scratch, no cleanup, no signal handler.

That also removed two things the scratch version needed and got wrong: the
INT/TERM trap deleted the workdir and returned, so the aggregation then
found no verdicts and printed an authoritative total outage; and five
SIGKILLed checks leaked five temp dirs, two of them still being written to by
jobs that outlived the parent.

The slow/hung counters still touch the disk, deliberately off the liveness
path: if they cannot be written the bound is off for that check and the
verdict says so, while every port's up/down is still what its probe found.
Verified on a live VM with the counter path pointed at something unwritable.

Three more defects the audit turned up, all fixed

A port configured twice was probed twice. Fanned out that is two jobs on
one counter file. Sequentially it was quieter and worse: the counter was
incremented twice per check, so the hung bound fired at half the
configured grace. With HEALTHCHECK_SLOW_GRACE=3 the base branch reports
DOWN hung on the second check. Each port is emitted once now.

The -w format reached curl with literal quotes around it. Every port on
both live VMs reported slow but listening while all 24 terminals were
answering 401. Every test in the file passed anyway, because the stub curl
ignores its arguments. Fixed, and the gap closed: a test now captures the real
argument vector and pins the format and --max-time. Live smoke is the only
thing that caught this.

The file's header said the container is alpine/ash. It is dash.

What was checked

Five phases, receipt kept locally under a gitignored directory, run against
exactly the three files this PR touches and nothing else.

Ten verification passes, an independent counter-review by someone with no
context on the change, live smoke on both production VMs, ten final passes.
Eight findings, six of them from the counter-review, two of which changed the
design. All fixed. One deferral: the dedupe silences a misconfiguration rather
than reporting it, and I have noted in the docs that config_helper.py is
where a duplicated port belongs, because a healthcheck's verdict is about
liveness.

Test defects found and fixed along the way: _run_with_slow_curl omitted
HEALTHCHECK_STATE_DIR and so read and wrote the real /tmp/healthcheck-slow,
and the dead-port test asserted a count of digit tokens where a duplicate plus
a drop would have passed. Added coverage for per-port verdict routing under
mixed replies, fail-closed on a job killed before it reports, the disk failure
above, and the curl argument vector.

make test-unit (518) and make lint are green. The suite runs the real
script under sh. The disk-failure test fails against both scratch-file
designs I tried, which is the point of it.

Merge order

This is stacked on #15, because that PR is what put the slow/hung bound into
the same loop; branching from master instead would have meant hand-resolving
the same file twice. It shows #15's commits until #15 lands, after which this
collapses to scripts/healthcheck.sh, its test file, and one docs section.

#15 → this → #21 → #16 → #18 → #19 → #10, and I will rebase
promptly as each predecessor merges. I rehearsed it: after #15, this and #21 both merge with zero conflicts, in either order, and the
combined tree is green. Happy to hold this until #15 is merged if you would
rather review it as a plain diff.

No CHANGELOG entry: the watchdog and this healthcheck work are all unreleased
and #15 carries no entry either, so a single entry covering the lot belongs on
whichever of these lands last. Say the word and I will add it there.

🤖 Generated with Claude Code

[Marinski]

Folding this into #15 rather than asking you to review it separately, and closing it here.

Seven open PRs from one person is a lot to put in front of you, and three of them were the same subject: #15 changes healthcheck.sh and vm-watchdog.py, and so did this one and #21. Splitting one feature across three reviews makes your job harder, not easier.

The commit is now on feat/vm-health-watchdog as 56a0ef5, unchanged. Nothing is lost and nothing is hidden: #15's description has a section describing exactly this, and the full reasoning above stays here for reference.

If you would rather review it on its own after all, say so and I will reopen it — the branch is untouched.