Skip to content

fix(cuda): isolate concurrent hardware floor probes - #20

Merged
ualtinok merged 1 commit into
cortexkit:masterfrom
Qiiks:fix/cuda-probe-cache-lock
Sep 17, 2026
Merged

ualtinok merged 1 commit into
cortexkit:masterfrom
Qiiks:fix/cuda-probe-cache-lock

Conversation

@Qiiks

@Qiiks Qiiks commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Release the global probe-cache mutex before running a worker child; per-worker Arc<OnceLock<_>> entries preserve one probe per worker and deliberate success/failure caching.
  • Add a regression where a worker confirms it is stalled; another worker must complete before the first is released.
  • Document the Linux ELF cuBLASLt runtime dependency requested in Windows owned-CUDA worker: delay-loaded cuBLASLt + sidecar packaging gate #18.

Follow-up to #18 discussion_r4037256258. The separate certification worker-path finding is already addressed in merged master and is not changed here. Based directly on current master; excludes the separate batch-advice change.

Verification

  • Old global-lock implementation: regression fails with unrelated probe blocked behind stalled worker: Timeout.
  • Fixed implementation, including after rebase: cargo test -p synapse-module --lib cuda_floor_probe -j 1 — 3 passed, 1 explicitly ignored hardware test.
  • cargo clippy -p synapse-module --lib -j 1 -- -D warnings and cargo fmt --all -- --check pass.
  • No GPU benchmark or deployment change; running native consumers remain untouched.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Previously, the global CUDA probe-cache lock was held while a worker child ran, so a stalled hardware probe blocked unrelated probes. The lock is now released before running the child; per-worker Arc<OnceLock<_>> entries preserve one probe and deliberate success/failure caching per worker.

  • Adds a regression test where a second worker completes while the first is still stalled.
  • Documents the Linux ELF cuBLASLt runtime dependency: missing libcublasLt.so.12 makes --probe-floor exit 127 before the driver-only probe.
  • No GPU benchmark or deployment change; running native consumers remain untouched.

Written for commit 9f9f5ec. Summary will update on new commits.

Review in cubic

Copilot AI lite review requested due to automatic review settings September 17, 2026 19:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Re-trigger cubic

@synapse-alfonso

Copy link
Copy Markdown

Verified beyond the description and merging. The regression test is real: I spliced master's probe body under this branch's tests and it fails exactly as you reported — unrelated probe blocked behind stalled worker: Timeout — then passes with the Arc<OnceLock> entry. Three probe tests green locally, clippy clean, CI green, Cargo.lock untouched. The shape is right: the map lock lives only as long as the entry lookup, the child runs outside it, and per-worker serialisation plus failure caching survive unchanged.

Landing through the CI train now with the Linux DT_NEEDED note you added, which closes the packaging asymmetry from the #17/#18 hardware pass.

@ualtinok
ualtinok merged commit 5e4db55 into cortexkit:master Sep 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants