Windows owned-CUDA worker: delay-loaded cuBLASLt + sidecar packaging gate

[Qiiks]

Closes #14 (item 1: delay-load + sidecar packaging).

What changed

• Delay-load cuBLASLt in ck-synapse-worker-cuda (MSVC+CUDA only, crates/synapse-worker-cuda/build.rs): /DELAYLOAD:cublasLt64_13.dll + delayimp. Verified with dumpbin /dependents that cudart/cublas are statically linked and cuBLASLt is the only delay-loaded DLL. nvcuda.dll/cudart64_13.dll/cublas64_13.dll were dropped from the DELOAD list after the linker reported "no imports found".
• Explicit runtime-library resolution (crates/synapse-engine-cuda/src/cuda.rs): before cuInit/cudaGetDevice, LoadLibraryW the four CUDA DLLs (LazyLock, MSVC-only). A missing library is now a Rust error naming the DLL instead of an unhandled loader exception (exit 53, no output).
• Sidecar packager scripts/package-owned-cuda.ps1: worker EXE + runtime DLLs at ZIP root, component licenses, manifest.json with SHA-256 hashes and source component attribution. Driver deliberately not bundled.
• Package verifier scripts/test-owned-cuda-package.ps1: extracts a fresh copy, verifies hashes, then asserts no-sidecar --version exit 0, missing-DLL --probe-floor non-zero with a named-DLL error, and side-by-side resolution with CUDA removed from PATH. -RequireGpu additionally requires a real floor JSON.
• Manual CI gate (tests.yml, windows-owned-cuda-manual): derives sidecars from the same pinned cuda_cudart/libcublas archives used for compilation (no independently maintained DLL manifest) and runs the verifier. GPU execution itself requires a GPU runner; the gate asserts loader correctness, not model execution.
• The old off-PATH discriminator in the build gate remains valid: with delay-load, its "0xC0000135 or fail" logic is superseded by the new package verifier, which asserts the two states the delay-loaded build actually has. Left untouched for the llama-CPU gate.

Verification (local RTX 4050, CUDA removed from PATH)

• Freshly extracted ZIP, no sidecars: --version → exit 0; --probe-floor → exit 1, Error: cannot load CUDA library cublasLt64_13.dll: The specified module could not be found. (os error 126).
• Same ZIP with sidecars: --probe-floor → exit 0, {"driver_api":13030,"compute_capability":{"major":8,"minor":9}}.
• cargo clippy -p synapse-engine-cuda -p synapse-worker-cuda --all-targets -D warnings clean; cargo fmt --check clean.

Non-goals

• Release-matrix publication (issue The owned-CUDA worker's packaging contract: no release asset, DLL-missing exe fails the acceptance probe at PE load, and no worker_bin default #14 item 2, alfonso's).
• Model-weight loading / embedding parity — unchanged from prior PRs.
• Linux/macOS: unaffected (MSVC-only guard).

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Closes #14 (item 1). Windows CUDA workers now delay-load cuBLASLt and run with packaged runtime sidecars, so --version works without CUDA on PATH while CUDA failures report the missing library instead of crashing during load. The module probes the worker for its hardware floor before creation, unless a complete environment override is available.

Packaging

• Requires CUDA 13 and derives DLLs from the pinned runtime components used to build the worker.
• Includes component licenses and SHA-256 metadata, rejects duplicate runtime filenames, and leaves the NVIDIA driver system-installed.
• Verifies no-sidecar startup, actionable missing-DLL refusal, and side-by-side resolution with CUDA removed from PATH; -RequireGpu additionally enforces the driver and compute-capability floor.

Hardware probe

• Selects the configured worker, engine override, or sibling binary, then caches each result—including failures—by canonical worker path.
• Applies driver API and compute-capability overrides only when both values parse successfully; otherwise it uses the probe.
• Bounds the subprocess to 10 seconds, limits stdout, and preserves stderr diagnostics without fabricating hardware readings.

^{Written for commit 6ee5494. Summary will update on new commits.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[cubic-dev-ai]

All reported issues were addressed across 9 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[synapse-alfonso]

Verified on a dispatched run of the manual gate against 10ec3fc8: archive hashes, no-DLL --version, the missing-DLL refusal naming the library, and side-by-side runtime resolution all pass on a real Windows runner. The new gate is stronger than the one it replaces — the hollow-green discriminator is now a named DLL rather than a status code. 4620d1b's -RequireGpu floor enforcement and license-path preservation read correctly.

One thing before merge: this branch is not rebased on #17's current head. ce87e1be on #17 fixed a vacuous real-worker test and bounded the probe's pipe reads by the deadline, and #18 still carries the earlier versions of those hunks in crates/synapse-module/src/lib.rs. Merging #17 then #18 in that state would either conflict or regress the fix. Please rebase #18 onto #17's head (and, as noted on #17, keep Cargo.lock at origin/master's bytes). I will re-dispatch the packaging gate on the rebased head before landing; the earlier dispatch already proves the packaging half, so that is a confirmation run, not a new question.

Merge order once both are ready: #17, then #18, then #16, on one train.

[Qiiks]

Rebased onto PR #17 head 3e47fa9; PR #18 is now d558d3e. Cargo.lock is restored to origin/master bytes in both branches. The corrected probe deadline and explicit real-worker regression are included. Windows worker builds now explicitly reject non-CUDA-13 toolkits because the package uses CUDA 13 DLL names; the CUDA 13 build passed locally. Ready for your packaging-gate confirmation run.

[Qiiks]

Pushed 6ee5494: hardware probe results and failure evidence are now keyed by canonical worker path. The regression runs two separate worker commands, verifies a failed package cannot contaminate a working package, and verifies failure caching stays scoped to that worker. Targeted regression passed locally. PR ancestry still includes #17 head 3e47fa9; catalog model.load repair is on a separate branch.

[cubic-dev-ai]

2 issues found across 1 file (changes from recent commits).

_{You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/synapse-module/src/lib.rs">

<violation number="1" location="crates/synapse-module/src/lib.rs:6050">
P2: When probes for different worker binaries overlap, this global mutex remains locked for the child-process timeout, serializing unrelated probes and synchronously blocking async probe tasks for up to ten seconds. Release the mutex before running the child, or use per-worker in-flight state so only callers for the same path wait.</violation>

<violation number="2" location="crates/synapse-module/src/lib.rs:6167">
P2: When a CUDA model specifies an explicit `worker_bin` and the sibling or environment worker differs or is absent, certification evidence probes that other path instead of the worker that was loaded. Preserve the worker path or successful reading from model loading and use it when generating CUDA evidence.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When a CUDA model specifies an explicit worker_bin and the sibling or environment worker differs or is absent, certification evidence probes that other path instead of the worker that was loaded. Preserve the worker path or successful reading from model loading and use it when generating CUDA evidence.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At crates/synapse-module/src/lib.rs, line 6167:

<comment>When a CUDA model specifies an explicit `worker_bin` and the sibling or environment worker differs or is absent, certification evidence probes that other path instead of the worker that was loaded. Preserve the worker path or successful reading from model loading and use it when generating CUDA evidence.</comment>

<file context>
@@ -6151,14 +6161,14 @@ fn parse_compute_capability(value: &str) -> Option<(u32, u32)> {
+fn owned_cuda_floor_observed(decision: &CudaFloorDecision, worker: Option<&Path>) -> Value {
+    let error = match decision {
+        CudaFloorDecision::Unsupported { observed: None, .. } => {
+            owned_cuda_probe_floor(worker).err()
+        }
+        _ => None,
</file context>

[cubic-dev-ai]

P2: When probes for different worker binaries overlap, this global mutex remains locked for the child-process timeout, serializing unrelated probes and synchronously blocking async probe tasks for up to ten seconds. Release the mutex before running the child, or use per-worker in-flight state so only callers for the same path wait.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At crates/synapse-module/src/lib.rs, line 6050:

<comment>When probes for different worker binaries overlap, this global mutex remains locked for the child-process timeout, serializing unrelated probes and synchronously blocking async probe tasks for up to ten seconds. Release the mutex before running the child, or use per-worker in-flight state so only callers for the same path wait.</comment>

<file context>
@@ -6035,20 +6035,30 @@ struct OwnedCudaFloorReading {
+        .or_else(|| resolve_worker_binary_sibling(CUDA_WORKER_ENGINE))
+        .ok_or_else(|| "CUDA floor probe worker binary not found".to_string())?;
+    let worker = fs::canonicalize(&worker).unwrap_or(worker);
+    let mut cache = OWNED_CUDA_PROBE
+        .lock()
+        .unwrap_or_else(|poisoned| poisoned.into_inner());
</file context>

[synapse-alfonso]

All three are merged locally on one branch in the order #17 → #18 → #16, verified with --locked against the sibling pins, and landing through the CI train now. The hardware pass covering them is in docs/audits/pr15-cuda-verification.md (RTX 4090, $1.22 total):

• --probe-floor printed {"driver_api":13000,"compute_capability":{"major":8,"minor":9}} in 442 ms, matching nvidia-smi (580.105.08 / CUDA 13.0 / CC 8.9) exactly. CUDA_VISIBLE_DEVICES="" refused on cuInit status 100 with no fabricated reading. The module admitted and certified Qwen3-Embedding on real hardware with the observed triple in evidence, and refused before worker creation under a 7.4 override with owned_cuda_unsupported carrying the observed values.
• fix(engine-cuda): bound Qwen3 shape-plan cache with a 2-entry LRU #16: eviction observed in the port's own arena log and in nvidia-smi residency (−100 MiB), bound held at two arenas across four shapes, the evicted 64-row plan rebuilt on the fourth forward, and its output byte-identical to the first forward and to a master-built worker.
• One fact for The owned-CUDA worker's packaging contract: no release asset, DLL-missing exe fails the acceptance probe at PE load, and no worker_bin default #14's follow-on, found on the way: on Linux the ELF worker keeps DT_NEEDED on libcublasLt.so.12, so --probe-floor there exits 127 without the runtime on the library path even though the probe needs only the driver API. The Windows delay-load has no Linux twin; worth a sentence in the README's packaging section when that lands.

Thanks for the lock restore and the rebase — both landed exactly as asked, and the two extra commits on #18 (per-worker-path probe cache, CUDA-13 toolkit guard) read correctly. The packaging gate was re-dispatched on the rebased head as a confirmation run.