fix(cuda): isolate concurrent hardware floor probes - #20
Conversation
There was a problem hiding this comment.
No issues found across 2 files
You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
|
Verified beyond the description and merging. The regression test is real: I spliced master's probe body under this branch's tests and it fails exactly as you reported — Landing through the CI train now with the Linux |
Summary
Follow-up to #18 discussion_r4037256258. The separate certification worker-path finding is already addressed in merged master and is not changed here. Based directly on current master; excludes the separate batch-advice change.
Verification
unrelated probe blocked behind stalled worker: Timeout.cargo test -p synapse-module --lib cuda_floor_probe -j 1— 3 passed, 1 explicitly ignored hardware test.cargo clippy -p synapse-module --lib -j 1 -- -D warningsandcargo fmt --all -- --checkpass.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by cubic
Previously, the global CUDA probe-cache lock was held while a worker child ran, so a stalled hardware probe blocked unrelated probes. The lock is now released before running the child; per-worker
Arc<OnceLock<_>>entries preserve one probe and deliberate success/failure caching per worker.cuBLASLtruntime dependency: missinglibcublasLt.so.12makes--probe-floorexit 127 before the driver-only probe.Written for commit 9f9f5ec. Summary will update on new commits.