The owned-CUDA capability floor is env-only with no probe and no docs — a supported machine refuses until the operator guesses two variable names

[Qiiks]

The owned-CUDA capability floor is read exclusively from environment variables that nothing produces:

// crates/synapse-module/src/lib.rs:5925
fn owned_cuda_floor_decision() -> CudaFloorDecision {
    let driver_api = ["SYNAPSE_CUDA_DRIVER_API", "CUDA_DRIVER_API"] ...
    let compute = ["SYNAPSE_CUDA_COMPUTE_CAPABILITY", "CUDA_COMPUTE_CAPABILITY"] ...
    let (Some(driver_api), Some((major, minor))) = (driver_api, compute) else {
        return CudaFloorDecision::Unsupported {
            reason: CudaUnsupportedReason::HardwareUnavailable, observed: None,
        };
    };

If either variable is unset, the decision is HardwareUnavailable — backend_unavailable before any worker is spawned, with no observed values in the refusal. There is no probe anywhere that fills these in, so a machine that comfortably passes the floor (RTX 4050, CC 8.9, driver API 13030 here) refuses every owned-CUDA load until the operator finds the variable names in source. They are documented nowhere: SYNAPSE_CUDA_DRIVER_API appears in exactly one place (the function above), and neither name appears in README or docs.

Two papercuts observed while setting this up

1. The driver value is the CUDA driver-API integer, not the marketing version. This box's NVIDIA driver is 610.88, but the floor compares against OWNED_CUDA_MINIMUM_DRIVER_API = 12040, so the correct value is 13030 (cuDriverGetVersion() — verified by calling it directly). Nothing in the env-var name or the refusal says which of the two numbers is expected; an operator who sets 610.88 silently parses to None (it is not a u32) and gets the same HardwareUnavailable as setting nothing.
2. The refusal drops the observed hardware. HardwareUnavailable carries observed: None, so the error cannot say "your CC is 8.9, the floor is 7.5, but the driver API was missing" — it cannot distinguish "no hardware" from "hardware fine, env incomplete".

Why this is a fixable gap, not an inherent one

The engine already talks to the CUDA driver directly (cuInit, cuCtxGetDevice, cuDevicePrimaryCtxRetain in crates/synapse-engine-cuda/src/cuda.rs), and device_meets_floor(driver_api, major, minor) already exists in synapse-engine-cuda/src/lib.rs:188. The two missing reads are cuDriverGetVersion and cuDeviceGetAttribute(..., COMPUTE_CAPABILITY_MAJOR/MINOR) — the same FFI pattern, a handful of lines.

Since the module deliberately does not link the CUDA engine (the worker is a separate process), the probe probably belongs on the worker side: the worker already reports machine facts in PONG (rss_mb, models_loaded), and the module could fold a worker-reported driver/CC pair into the floor decision (or at minimum into the refusal's observed). Design choice yours; the issue is that today there is no path at all.

Proposed behavior

• Keep env vars as an override.
• When unset, probe: cuDriverGetVersion + device attributes, and cache the result per process.
• Unsupported refusals carry observed whenever the probe ran, so the message distinguishes below-floor from missing-input.
• Document both variables (and the API-integer format) in README next to the CUDA section.

Happy to implement if the shape above looks right — I can't run the CUDA build gate locally, so I'd want your steer on whether the probe lives in the worker or the module before writing it.

[synapse-alfonso]

Confirmed at source: owned_cuda_floor_decision (crates/synapse-module/src/lib.rs:5925) reads only the two env pairs and returns HardwareUnavailable { observed: None } when either is unset, and nothing in the tree populates them. Both papercuts hold as described — the driver value is the CUDA driver-API integer, and the refusal drops whatever was observed.

Steer on where the probe lives: the worker side, as a short-lived invocation rather than the HELLO/PONG path.

• The module deliberately does not link the CUDA driver, and the floor decision runs before any worker is spawned. Reporting through PONG would move the gate after spawn, which changes the contract for a refusal that today never starts a process.
• A driver probe is the kind of call that can fault on a broken install, and the worker is the crash domain. A ck-synapse-worker-cuda --probe-floor subcommand that prints { "driver_api": 13030, "compute_capability": { "major": 8, "minor": 9 } } keeps a segfault in a child; the module runs it once per process with a timeout, caches the result, and treats a non-zero exit or unparseable output as HardwareUnavailable with the stderr tail as observed context.
• Env vars stay as the override and are checked first, so a rig can still pin the decision without a working driver.
• Unsupported carries observed whenever the probe produced values, so the message distinguishes below-floor from missing-input.
• README gets both variables and the API-integer format next to the CUDA section.

Happy to take that as a PR. The windows-owned-cuda-manual-gate from #10 now runs green on dispatch, so I can exercise the probe against the real toolchain before merging.

[Qiiks]

Agreed on all four points — the worker-side --probe-floor subcommand is the right shape, and for the reason you give: the module must not link the CUDA driver and the decision has to happen before spawn, so a probe that runs in the worker and reports back over PONG would move the gate after the thing it gates.

Plan as I'll build it:

1. ck-synapse-worker-cuda --probe-floor (hidden from help like --test-abort*) prints exactly one JSON object to stdout and exits 0:
   {"driver_api": 13030, "compute_capability": {"major": 8, "minor": 9}}
   It links the driver only for this path. A second --probe-floor run is not needed — the module calls it once per process and caches.
2. owned_cuda_floor_decision keeps the env pair as the first check (override wins, as today), and only when a value is unset does it consult the cached probe.
3. Module runs the probe once, with a timeout, treating non-zero exit or unparseable stdout as HardwareUnavailable { observed: Some(stderr_tail) } instead of the current None.
4. Unsupported carries observed whenever the probe produced values, so below-floor and missing-input read differently.
5. README gets SYNAPSE_CUDA_DRIVER_API and SYNAPSE_CUDA_COMPUTE_CAPABILITY next to the CUDA section with the driver-API-integer note (the value is cuDriverGetVersion(), not the marketing string — the local live probe returned 13030 on driver 610.88, which is what a naive driver-version parse would have rejected).

Taking it as a PR. Since windows-owned-cuda-manual-gate now runs green on dispatch, I'll exercise the probe against the real toolchain before opening, and I'll link the run here.