Skip to content

Ci/cuda test matrix smoke - #7757

Draft
MrLi000001 wants to merge 3 commits into
deepmodeling:developfrom
MrLi000001:ci/cuda-test-matrix-smoke
Draft

Ci/cuda test matrix smoke#7757
MrLi000001 wants to merge 3 commits into
deepmodeling:developfrom
MrLi000001:ci/cuda-test-matrix-smoke

Conversation

@MrLi000001

@MrLi000001 MrLi000001 commented Aug 2, 2026 •
edited
Loading

Copy link
Copy Markdown

Split the CUDA Test workflow into a build job (builds once and uploads the binary as an artifact) plus a matrix of GPU test jobs, so the six test suites run in parallel on PRs instead of serially.

  • pull_request runs the 01_PW smoke subset (new CASES_GPU_SMOKE.txt, 23 representative cases covering solvers, nspin=1/2/4, relax, symmetry, vdW, MD, DFT+half, output options); schedule (nightly) and workflow_dispatch run the full CASES_GPU.txt.
  • Fix (review): the workflow-level concurrency.group referenced matrix.suite, which is unavailable at workflow scope and made the workflow fail to run. The group is now workflow+ref only.
  • Fix (review): 01_PW GPU tests stay at -n 1 (add 01_PW to GPU CI test suite #7690 intentionally used single-process on the single-GPU CI pods); the win comes from the smoke subset, not from MPI process count.
  • ccache statistics are printed in every run (30G limit, zero-stats, before/after).

Verification

  • Prior serial CUDA Test: 6 suites ~9 min of tests (01_PW 73 cases alone ~4.8 min) behind one build.
  • This PR: 01_PW PR runs drop from 73 to 23 cases; suites run in parallel. Re-measure end-to-end after the concurrency fix (container init ~8-10 min each is the cost to watch; if the parallel matrix costs more than it saves, group the short suites 12/13/15/16 into one job).

Reminder

  • I have read AGENTS.md and docs/developers_guide/agent_governance.md.
  • I have linked an issue or explained why this PR does not need one.
  • I have added adequate unit tests and/or case tests, or explained why not.
  • I have listed the exact verification commands run and their results.
  • I have described user-visible behavior changes, including INPUT parameter changes.
  • I have explained core-module impact for ESolver, HSolver, ElecState, Hamilt, Operator, Psi, or other source/ changes.
  • I have requested any needed governance exception below.

Linked Issue

Fix #

Unit Tests and/or Case Tests for my changes

  • Commands run:
  • Result summary:
  • Checks not run, with reason:

What's changed?

  • Example: brief summary of the user-visible or developer-facing change.

Governance Notes

  • INPUT/docs changes:
  • Core module impact:
  • Exceptions requested:

@MrLi000001
ci(cuda): build only for CI runner GPU arch and raise parallelism
e614a28 
The CUDA CI built every .cu file for 7 GPU architectures
(60/70/75/80/86/89/90) with a hardcoded -j4, so the Configure & Build
step took ~33 min even with a warm ccache.

- Pin CMAKE_CUDA_ARCHITECTURES=70: the CI GPU pool is Tesla V100
  (sm_70, per nvidia-smi in the run logs and the '16V100' Slurm
  partition in .ci/slurm/config.ini). This cuts nvcc work by ~7x.
- Build with -j $(nproc) instead of -j4; with the arch list reduced,
  the higher parallelism is memory-safe.

Expected: Configure & Build ~33 min -> ~10 min on a cache-cold run.
@MrLi000001
ci(cuda): split GPU tests into matrix jobs and add 01_PW smoke list
bb4157b 
The single-job CUDA workflow ran the build and all GPU test suites
serially. After deepmodeling#7690 added 73 01_PW cases executed with -n 1, the GPU
test portion grew to ~9 min serial on top of the build.

- Split into a build job (publishes the installed binary as an
  artifact) and a matrix of 6 gpu-test jobs that download the binary
  and run their suites in parallel.
- 01_PW runs with -n 2 like the other suites, and pull requests now run
  a new 23-case smoke subset (CASES_GPU_SMOKE.txt, covering solvers,
  spin variants, smearing/mixing, relax, symmetry, vdW, MD, DFT+half
  and output options). The full CASES_GPU.txt list runs nightly via the
  new schedule trigger and on workflow_dispatch.
- Add ccache sizing/statistics steps for observability.

Expected: end-to-end CUDA CI wall time roughly halves; the 01_PW step
drops from ~4.8 min serial to ~2 min within its own parallel job.
@mohanchen mohanchen added GPU & DCU & HPC GPU and DCU and HPC related any issues Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS Refactor Refactor ABACUS codes labels Aug 2, 2026
@Stardust0831

Stardust0831 commented Aug 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The workflow-level concurrency expression references matrix.suite, but the matrix context is not available at workflow scope, so this head produced no CUDA Test run. After fixing that, please reconsider the matrix design: one GPU container initialization took 8m39s, while all serial GPU tests took about 9 minutes, so six containers may increase queueing and GPU usage. Reducing the PR 01_PW coverage from 73 cases to 23 also needs an explicit coverage policy.

@MrLi000001
MrLi000001 marked this pull request as draft August 3, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS GPU & DCU & HPC GPU and DCU and HPC related any issues Refactor Refactor ABACUS codes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@MrLi000001 @Stardust0831 @mohanchen