Fix profiler and TAU harness checks - #181
Open
dannybaths wants to merge 1 commit into
Open
Conversation
Make profiler and TAU tests assert real artifacts instead of matching diagnostic text, fail TAU builds explicitly, detect MPI by querying the wrapper that will actually be used, and check the rocprof-compute dependency pins before a long profiling run rather than after it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes several profiler and TAU test harness issues found while validating the AAC7 OpenMPI row.
corresponding CTest pass expressions are updated to match.
ompi_infoexists somewhere on
PATH.directory.
analyzeenforces before profiling,and the rocprof-compute tests fail immediately, with the reason, when Python is older than 3.10.
These are reported as failures rather than skips deliberately, so the failure count is not quietly
reduced.
Why
Several tests could pass or fail for the wrong reason:
profile.0is a substring ofCould not open profile.0.0.0, andprotoappears in output the tool prints when it producesnothing.
makefailed, so the final verdict depended only on regex output.Because
PASS_REGULAR_EXPRESSIONoverrides the exit code, the script and the CMake criteria had tobe fixed together.
ompi_infowas visible butmpic++hadbeen replaced by MPICH wrappers.
analyzestep thatcould not run, turning a dependency problem into a timeout that named no cause.
Validation
Validated on AAC7 with the OpenMPI PrgEnv row: MI300A, ROCm 7.14.0, OpenMPI 5.0.10 over OFI/CXI.
pass:
Rocprof-sys_ROCm_Stream_Overlap,Rocprof-compute_ROCm_Roofline_Check,TAU_Trace_Check,TAU_Profile_Check,TAU_Check_HIP_ProfileandTAU_Check_MPI_Profile. Each now matches on astring only the success path can produce.
test fails in about three seconds, reporting the version it found and the version it needs. Under
Python 3.12.12 it runs to completion and the roofline assertion finds a real
roofline.csv. Thefailing case is deliberately a failure rather than a skip, so an environment that cannot run
rocprof-compute is not quietly removed from the failure count.
tests build and produce real
profile.*/tautrace.*artifacts andpprofoutput containing MPIand HIP routines.
node-local
/tmp, but rank 1 was launched on the second node where that directory did not exist. ThisPR fixes that launch assumption too, by keeping the ranks on the node that owns the build directory.
Verified on two nodes with the checkout in node-local
/tmp— the failure reproduces without the fixand both invocations pass with it — and then in the full two-node suite, where all four Jacobi TAU
tests pass on real artifacts (319 pass / 27 fail / 15 skip / 1 timeout of 362, suspicious-pass 0).
the OpenMPI-built
rccl-testsbenchmark. This PR does not attempt to solve that packaging mismatch.harness bug.