ci: take gpu-l40s out of rotation and seed frontier10202 as excluded - #1804
Open
sbryngelson wants to merge 1 commit into
Open
ci: take gpu-l40s out of rotation and seed frontier10202 as excluded#1804sbryngelson wants to merge 1 commit into
sbryngelson wants to merge 1 commit into
Conversation
Two GPU resources accounted for most of the non-code failures in run 33553417354. gpu-l40s has been failing jobs for weeks, and it is also what the partition selector kept choosing and then starving on: four Phoenix lanes submitted, never started, and were cancelled after 3-5.5h. Two causes, both fixed here. The selector counted "mix" nodes as available, but a mix node is partially allocated and may have no free GPU -- it picked gpu-l40s on "1 idle/mix nodes" and queued until timeout -- so only fully idle nodes count now. And l40s is out of the priority list, fallback included: leaving the fallback pointed at it sent jobs back there whenever nothing was idle. frontier10202 produced all 183 GPU memory-access faults in that run, across 43 distinct tests, three of which exhausted their retries; the same lanes passed on eight other g1 nodes with zero faults. It is seeded into node_exclude the way phoenix already seeds its two nodes, so the preflight from MFlowCode#1797 can still add more at run time. Its faults are intermittent (379 of 382 tests passed there), which is why syscheck can clear it and a seed is still worth having. Drop it once OLCF has looked at the node. Claude-Session: https://claude.ai/code/session_01XZjDp6ch827LVJRrvVb8hC
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
This PR updates CI Slurm GPU selection and exclusion behavior to reduce non-code test failures caused by flaky GPU resources.
Changes:
- Remove
gpu-l40sfrom GPU partition priority and change fallback togpu-a100. - Count only fully
idlenodes (notmix) when determining GPU-partition availability. - Seed
frontier10202into the node exclude list for the relevant GPU cluster.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| .github/scripts/submit-slurm-job.sh | Seeds frontier10202 into node_exclude to avoid a flaky node while still allowing runtime preflight appends. |
| .github/scripts/select-gpu-partition.sh | Adjusts GPU partition selection logic (priority/fallback) and tightens “idle node” counting/logging. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| SELECTED_GPU_PARTITION="" | ||
| for _part in $_GPU_PARTITION_PRIORITY; do | ||
| _idle=$(sinfo -p "$_part" --noheader -o "%t" 2>/dev/null | grep -cE "^(idle|mix)" || true) | ||
| _idle=$(sinfo -p "$_part" --noheader -o "%t" 2>/dev/null | grep -cE "^idle" || true) |
| SELECTED_GPU_PARTITION="" | ||
| for _part in $_GPU_PARTITION_PRIORITY; do | ||
| _idle=$(sinfo -p "$_part" --noheader -o "%t" 2>/dev/null | grep -cE "^(idle|mix)" || true) | ||
| _idle=$(sinfo -p "$_part" --noheader -o "%t" 2>/dev/null | grep -cE "^idle" || true) |
| # in run 33553417354 (43 distinct tests) while the same lanes passed | ||
| # on eight other g1 nodes with none. Its faults are intermittent -- | ||
| # 379 of 382 tests still passed there -- so syscheck can clear it. | ||
| node_exclude="frontier10202" |
|
Claude Code Review Head SHA: 4bb63a4 Files changed:
Findings:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1804 +/- ##
=======================================
Coverage 61.69% 61.69%
=======================================
Files 84 84
Lines 21620 21620
Branches 3196 3196
=======================================
Hits 13338 13338
Misses 6090 6090
Partials 2192 2192 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two GPU resources accounted for most of the non-code failures in run 33553417354. Both are addressed here; neither touches solver code.
gpu-l40sout of rotationIt has been failing jobs for weeks, and it is also the partition the selector kept choosing and then starving on — four Phoenix lanes submitted, never started, cancelled after 3–5.5 h.
Two independent causes, both fixed:
mixcounted as available. Amixnode is partially allocated and may have no free GPU. CI loggedSelected GPU partition: gpu-l40s (1 idle/mix nodes)and then queued until the job timeout. Only fully idle nodes count now.gpu-a100.Priority is now
gpu-h200 gpu-h100 gpu-a100 gpu-v100.frontier10202seeded into the exclude listThat node produced all 183 GPU memory-access faults in the run, across 43 distinct tests; three exhausted their retries (
851F7AE2,12ECE133,05A8C23C). The same lanes passed onfrontier10181/82/93/97/10203/06/07/10with zero faults, and the same tests passed undergpu-accon other nodes in that very run.It is seeded into
node_excludeexactly the way phoenix already seeds its two nodes, so the preflight added in #1797 can still append more at run time — this does not replace or bypass that mechanism.Its faults are intermittent (379 of 382 tests still passed on it), which is why
syscheckcan clear the node and a static seed is still worth having. Drop it once OLCF has looked at the node.Not covered here
The third failure class in that run was the self-hosted runner
frontier-22losing communication with GitHub (three jobs, no steps executed). That is a runner-host problem, not something the submit scripts can influence.Verification
Both scripts pass
bash -n.submit-slurm-job.shstill renders exactly one#SBATCH --exclude=directive (the dynamic one) — the seed feeds it rather than competing with it.Fallback choice is worth a second opinion: I picked
gpu-a100without knowing which Phoenix partition has the most nodes.