fix(daemon): name errno/path on listen failure and fail the client fast instead of a 30s timeout (#1828) - #2113
Open
DeusData wants to merge 4 commits into
Open
fix(daemon): name errno/path on listen failure and fail the client fast instead of a 30s timeout (#1828)#2113DeusData wants to merge 4 commits into
DeusData wants to merge 4 commits into
Conversation
A full /tmp made every daemon start die at pending publication, and the only durable trace was `daemon.ipc.listen_failed stage=pending_publication` -- no syscall, no errno, no path. The reporter needed hours (and a `df` on the wrong mount) to find the cause (#1828). Every listener failure now records the stage, the errno the failing step reported, and the artifact path it was operating on, and the log line carries all three: daemon.ipc.listen_failed stage=pending_publication errno=ENOSPC path=/tmp/cbm-daemon-1000/cbm-<key>.pending.tmp The record publication helper notes which step refused (create, write, link, unlink, sync, close, read-back) and on which artifact, so the listener can name it. The bind failure keeps its errno but joins the same shape (stage=socket_bind errno=... path=...) instead of a bare numeric bind_errno. cbm_errno_name() maps the portable errno set to its symbol. cbm_daemon_ipc_listen_failure_detail() exposes the same triple to the daemon host for the fail-fast record that follows. Test: daemon_ipc_listen_failure_names_errno_and_path forces the record write to fail with ENOSPC through a narrow inert seam (the socket binds, the file is created, only its data write is refused -- the reporter's exact shape) and asserts the log line names the errno and the temp path, and that the namespace is left clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A daemon that died at listener publication (a full /tmp) took the lifetime reservation, failed, and released it; the waiting client saw the generation vanish, respawned it, and repeated that for its whole 30 s deadline before reporting "CBM daemon is active or starting but could not accept this client" -- the opposite of what happened, with no cause and no path. MCP hosts then dropped the server silently (#1828). A detached daemon has no stderr and no channel back to its launcher, so the host now appends one start-failure record per failed start to <cache>/logs/cbm-daemon.start-failures.log -- next to cbm-daemon.log, through the same owner-only private-log open. Deliberately NOT the runtime directory: that is exactly what a full /tmp cannot write, and a new artifact there would entangle the stale-generation cleanup that #1894 just hardened. The record carries the listener detail from the previous commit (component, stage, errno, path) plus the runtime directory and a wall-clock stamp, and is written while the lifetime reservation is still held, so a client that watches the generation vanish already finds the cause. The client bootstrap loop reads that record (every 50 ms after its own spawn, and unconditionally before any relaunch), accepts only records for its endpoint not older than its spawn instant (2 s skew), and stops at once: error: CBM daemon failed to start: pending_publication failed with ENOSPC (No space left on device) at /tmp/cbm-daemon-1000/cbm-<key>.sock.pending.tmp; see ~/.cache/codebase-memory-mcp/logs/cbm-daemon.log The read is an optional bootstrap op (unit contracts with fake ops are untouched); the production op resolves the cache log directory. Every other wait state -- a live or stopping generation, a mute holder, a concurrent starter -- keeps its existing semantics and diagnostics. Tests (daemon_bootstrap): - daemon_bootstrap_fails_fast_when_daemon_dies_at_publication: the production bootstrap (real cohort, probe, startup lock, handoff) with a seam-only spawn override that forks a REAL daemon host whose record writes fail with ENOSPC. RED before: 34.7 s and "active or starting"; GREEN after: FAILED in well under 5 s naming ENOSPC and the temp path, one child spawned, daemon log carries errno=ENOSPC path=... . - daemon_bootstrap_start_failure_record_round_trip: record/read/format contract, a stale record and another endpoint's record are ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…not see it unused The rich errno+path recorder ipc_listen_failed (and ipc_listen_failure_reset) are defined before the POSIX '#ifndef _WIN32' listener block but only called inside it, so the Windows build compiled them dead and failed under -Werror,-Wunused-function (pr-smoke windows-latest). Wrap both in the same '#ifndef _WIN32'; the failure struct and its cross-platform accessor cbm_daemon_ipc_listen_failure_detail stay compiled on every platform. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…ootstrap ENOSPC test (#2113, O9) The #1828 fail-fast test asserted `elapsed_ms < 5000`, a wall-clock budget that flaked on the ASan test-diag lane under CI contention while the fail-fast mechanism itself engaged correctly: the whole daemon_bootstrap suite ran in 11 s, so the test never reached the 30 s slow-timeout path -- it only overran the 5 s wall-clock budget. Per O9 a gate never asserts a transient timing window, and a rerun is not a fix. Drop the wall-clock measurement and rely on the deterministic mechanism the test already observes: the client surfaces the recorded ENOSPC cause verbatim ("failed to start" + errno + path), the slow "active or starting" timeout wording is absent, and exactly one daemon is spawned. That "failed to start" message is produced only on the fast-fail break (cbm_daemon_bootstrap_start_failure_format), never on the 30 s deadline path, so these assertions bind the fast-fail contract without any timing. Verified RED-on-revert: defeating the fast-fail detection makes the client hang to the 30 s deadline and report "active or starting", tripping ASSERT_FALSE(stale_wording). Full daemon_bootstrap suite green (28/28) under the ASan/UBSan build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1828 — a daemon that cannot start (e.g.
/tmpfull) reported a misleading 30 s "active or starting" timeout and no cause.Root cause: the listener's publish helper returned a bare
bool(errno and path dropped), and a daemon that failed at publication released its lifetime reservation with no signal to its launcher, so the client's wait loop could not tell "died" from "starting" and burned the full 30 s deadline before reporting the wrong thing.Two commits:
daemon.ipc.listen_failed(every stage now carriesstage= errno= path=), e.g.stage=pending_publication errno=ENOSPC path=/tmp/cbm-daemon-<uid>/….<cache>/logs/cbm-daemon.start-failures.log, owner-only, rotated) the client checks, so a doomed start surfaces asCBM daemon failed to start: <stage> failed with <errno> at <path>; see …/cbm-daemon.login about a second instead of a 30 s "active or starting" wait. fix: recover markerless stale rendezvous sockets during activation #1894's stale-generation recovery is preserved.ENOSPC is injected in tests through a narrow inert seam (the socket binds, only the record write fails) — the reporter's exact tmpfs-full shape, deterministic, no RAM disk. Local verification (macOS):
daemon_ipc daemon_runtime daemon_bootstrap daemon_application cli platform508/0;make lint-ciclean; RED→GREEN→RED-on-revert on both tests (step 2's RED shows the 34.7 s deadline burn, GREEN 6.6 s). Thestatvfspreflight is left as a recorded follow-up; a relocatable runtime dir remains a separate product decision.🤖 Generated with Claude Code