GPU fault diagnostics: attribution on all four lanes; CCE needs the ROCm debug agent

[sbryngelson]

Updated. The original version of this issue concluded that CCE faults could not be attributed without a rebuild. That was wrong — see the correction. This body is the current state; measured and inferred claims are marked separately throughout.

Summary

GPU memory faults in CI were reported as a bare device address. All four GPU lanes can now name the faulting kernel. #1800 ships this. What remains open is listed at the bottom.

Method

A known out-of-bounds device write was injected into s_tvd_rk (m_time_steppers.fpp, inside the GPU_PARALLEL_LOOP, under if (igr), #ifdef MFC_GPU):

q_cons_ts(1)%vf(i)%sf(j + 2000000000, k, l) = 1._wp

Ground truth is therefore known exactly: m_time_steppers.fpp:486, s_tvd_rk. Every claim marked measured below was checked against it on real CI hardware. This mattered — four mechanisms that looked correct turned out to do nothing, and each was only caught because the right answer was known in advance.

Attribution, per lane

lane	mechanism	status
NVHPC (Phoenix)	Function: s_tvd_rk:438, Line: 486 — free, attempt 1	measured
AFAR/OpenMP (frontier_amd)	Kernel 0: ..._QMm_time_steppersPs_tvd_rk @ 486, 189/189 — free, attempt 1	measured
CCE/OpenACC	ROCm debug agent → Disassembly for function s_tvd_rk$m_time_steppers_$ck_L486_6	measured
CCE/OpenMP	agent hooks ROCr, below both offload models, so it should fire — but the symbol above is CCE's OpenACC mangling, and whether module/subroutine/line survive in OpenMP-offload form is unknown	expected, untested

NVHPC and AFAR already printed the faulting kernel unaided on the first attempt. The premise the original design rested on — "a fault reports only an address, so spend a retry" — was false on both.

The real gap on NVHPC was different: is_gpu_memory_fault() matched only AMD's wording, so 189 faults on a Phoenix gpu-acc shard were never classified as GPU faults at all. NVHPC says Accelerator Fatal Error / CUDA_ERROR_ILLEGAL_ADDRESS.

CCE: what works and what doesn't

Works — the ROCm debug agent (HSA_TOOLS_LIB=librocm-debug-agent.so.2)

Measured on Frontier, CCE 19.0.0 / ROCm 6.3.1 / cpe/25.03, --gpu acc. Prints straight to the job log: the faulting kernel by name, the faulting instruction marked =>, per-wave scalar/vector/system/trap registers, and wave_N: pc=0x… (stopped, reason: MEMORY_VIOLATION). No recompile, no execution-model change, no core files, no post-mortem step.

Volume: ~14k lines for 125 faulting waves on a 49×39 case — one identical disassembly block and a 115-line register dump repeated per wave. summarize_rocm_debug_agent() collapses that to ~37 lines.

A fixed tail cannot substitute. On the real report the first 80 lines are one wave's registers and the last 80 are another's, and the kernel name appears in neither. The summary keeps the full stop-PC histogram because the waves halted at four distinct PCs whose modal one is a load while the injected fault is a write — quoting a single PC hands the reader the wrong instruction.

Doesn't work — CRAY_ACC_DEBUG

Deliberately not enabled. The last kernel logged before each fault was s_write_run_time_information 81 of 102 traced faults, and the true culprit 0. CCE defaults to acc_model=auto_async_kernel, so the fault surfaces at whatever sync point catches it — here m_data_output's readback for the progress line. It names the routine owning the sync point, not the bad write. AMD_SERIALIZE_KERNEL/COPY do not help: HIP variables, and CCE's runtime is not HIP.

-hacc_model=auto_async_none would fix it, but it is a compile flag, and shipping it would stop CI exercising the asynchronous dispatch production uses — a coverage loss traded for fault messages. Rejected.

Doesn't work in CI — GPU core dumps

HSA_ENABLE_DEBUG=1 produces a dump, but fallocate() is not implemented on Lustre. Same binary/case/node/fault, varying only cwd:

cwd	gpucore	LOAD segments	result
/tmp (tmpfs)	305 MB	21	complete
/dev/shm (tmpfs)	305 MB	21	complete
Lustre	1296 B	0	Failed to allocate file: Bad file descriptor

MFC runs tests in tests/<UUID>/ on Lustre, so that lane can never produce a usable dump where it runs. This explains GPU core dump failed in the original CI logs. Superseded by the debug agent, which needs no workaround — and the two are mutually exclusive (with the agent loaded, ROCr reports Failed to enable debug interface and writes no dump).

If anyone pursues dumps anyway: ulimit -c unlimited is mandatory (with ulimit -c 0 ROCr silently skips the GPU dump — no file, no error), and a gpucore alone is not debuggable — rocgdb needs roccoremerge merged.core core gpucore.<pid> first.

What #1800 ships

• Diagnostics on every run, not a retry. Both OFFLOAD_TRACK_* variables are inert until an abort, so withholding them only cost an extra run to learn what the first could have said. The diagnostic retry is removed.
• OFFLOAD_TRACK_ALLOCATION_TRACES (was the address ever a real allocation) and OFFLOAD_TRACK_NUM_KERNEL_LAUNCH_TRACES=8 (launch stack traces).
• HSA_TOOLS_LIB wherever the agent library is reachable, plus summarize_rocm_debug_agent().
• NVHPC signatures, and a distinct "GPU memory fault" bucket in classify_error.

The availability gate is evaluated per call, never at import. On Frontier the library is on disk the whole time but only reaches LD_LIBRARY_PATH once mfc.sh load runs — an import-time gate reports "absent" on the one machine this exists for, indistinguishably from Phoenix where it truly is missing. It probes for the file rather than dlopening it, so testing the subprocess's environment does not load a debug agent into the harness.

Open

1. CCE OpenMP is untested. Every command in the investigation was --gpu acc. One --gpu mp run with the injection moves the last cell from expected to measured — the question is the symbol form, not whether the agent fires.
2. The AFAR interaction is unverified. Both the agent and the OFFLOAD_TRACK_* variables are reachable on frontier_amd. The agent is mutually exclusive with ROCr core dumps, so it may likewise supersede libomptarget's own fault report. Worst realistic case is one working diagnostic replacing a more detailed one; if a real fault shows otherwise, gate HSA_TOOLS_LIB on the lane.
3. Agent overhead is n=1. One paired A/B: 4.5645 ns/gp/eq/rhs against an agent-free spread of 4.5301–4.5614 — 0.07% above a range 0.69% wide, i.e. inside the noise. The repeats were cancelled by decision, not measured. This is no effect detected, not no effect.
4. The summarizer's regexes are anchored to ROCm 6.3.1's format, observed on one fault shape. A format change degrades silently: no wave match → empty summary → fall back to 14k raw lines, with nothing saying why. A test pins the field set against the real report's output, but the raw log did not survive teardown, so the input fixture is a reconstruction.
5. bench.py and run_case_optimization.sh still have nothing. Both run GPU cases; neither classifies a fault or gets any of the above.

Method notes

Two failure modes account for nearly every dead end here, and both are worth naming:

• Checking that an artifact exists rather than that it contains anything. The original core-dump test checked find . -name 'gpucore.*' and scored any hit as success — but the Lustre stub is a valid AMD HSA core ELF with every program header NULL, so it passes a filename check while containing no GPU memory. Verify LOAD segments (readelf -l), not filenames. The same shape produced the original bug in DO NOT MERGE: diagnose a GPU memory fault on the retry that follows it #1800: tests asserted the source mentioned the right identifiers, so a marker written on one side and never read on the other passed seven of them.
• Inference presented as measurement. AMD_SERIALIZE_* was asserted from the mechanism and did nothing. "CCE cannot attribute" generalised from CCE's own trace to the machine and was wrong. The CCE OpenMP row above was an inference sitting in a table of results. Where a claim here is not measured, it says so.

[sbryngelson]

Correction: CCE faults can be attributed, env-only

The conclusion above — that CCE cannot name a faulting kernel without a rebuild — is wrong, and the measurements below supersede it. It was drawn from CCE's own trace (CRAY_ACC_DEBUG) and generalised from that to the machine. The information is available one layer down, at ROCr.

Measured on Frontier: CCE 19.0.0, ROCm 6.3.1, cpe/25.03, gpu-acc. Same injected fault, repro case AFBCBDFA, 1 rank.

The answer for CI: the ROCm debug agent

export HSA_TOOLS_LIB=librocm-debug-agent.so.2

Prints straight to the job log, and names the source location:

Disassembly for function s_tvd_rk$m_time_steppers_$ck_L486_6:

Subroutine s_tvd_rk, module m_time_steppers, source line 486 — the enclosing GPU_PARALLEL_LOOP of the injected write. Exactly the ground truth. Plus, per wave: the faulting instruction marked =>, full scalar and vector registers, system registers (mode/trapsts/xnack_mask/flat_scratch/status), trap registers, and wave_N: pc=0x… (stopped, reason: MEMORY_VIOLATION).

No recompile, no execution-model change, no Lustre workaround, no core files, no post-mortem step, no artifact to archive.

Caveats:

• Mutually exclusive with the core-dump path. With the agent loaded, ROCr reports Failed to enable debug interface, debugger might be already attached and produces no gpucore at all. Pick one.
• Volume: ~14k lines for 125 faulting waves on a 49x39 case, with the kernel disassembly header repeated per wave. CI will want this capped or filtered.
• Cost on a healthy run is unmeasured. That decides always-on for the CCE lane versus a documented recipe. Do not assume it is free.

GPU core dumps: real, but unusable on Lustre

HSA_ENABLE_DEBUG=1 does produce a dump, but fallocate() is not implemented on Lustre. ROCr logs Failed to allocate file: Bad file descriptor, then GPU core dump failed — and still leaves a gpucore.<pid> behind. Controlled A/B, same binary/case/node/fault, varying only cwd:

cwd	gpucore size	LOAD segments	result
/tmp (tmpfs)	305 MB	21	complete
/dev/shm (tmpfs)	305 MB	21	complete
Lustre	1296 B	0	Failed to allocate file: Bad file descriptor

This is a filesystem limitation, not a CCE or ROCr defect — and it explains the GPU core dump failed line that appears in the original CI logs. MFC runs tests in tests/<UUID>/ on Lustre, so that lane can never produce a usable dump where it currently runs; it would need cwd on node-local /tmp.

Two further gotchas if anyone pursues this path anyway:

• ulimit -c unlimited is mandatory. With ulimit -c 0, ROCr skips the GPU dump entirely and prints nothing — no file, no error, no warning. The gpucore is gated on the host RLIMIT_CORE.
• A gpucore alone is not debuggable. rocgdb <exe> <gpucore> gives Couldn't find general-purpose registers in core file. It must be merged with the host core first:

  roccoremerge merged.core core gpucore.<pid>   # /opt/rocm-6.3.1/bin
  rocgdb <exe> merged.core

  That works, yielding Program terminated with signal SIGABRT and ~47 AMDGPU Wave 1:4:1:N … stopped because of SIGSEGV threads.

Method note

The original test design checked find . -name 'gpucore.*' and treated any hit as success. That is a false positive on Lustre: the 1296-byte stub is a valid AMD HSA core ELF with every program header NULL, so it passes a filename check while containing no GPU memory at all. Verify LOAD segment count (readelf -l), not filename existence.

That mistake is the same one that produced three earlier dead ends in this work — checking that an artifact exists rather than that it contains anything. It is worth stating plainly because it is the failure mode this whole investigation kept hitting.

[sbryngelson]

Update: AFAR verified, one shipped bug found, open item 2 closed

The summarizer silently produced nothing on ROCm 7.2.0

The regexes were written against ROCm 6.3.1 and returned '' for 65,210 lines of real 7.2.0 agent output — on the AFAR lane, the one they were meant to serve, with no error to explain it. Two format changes between versions:

6.3.1  wave_124: pc=0x7ff77e253408 (stopped, reason: MEMORY_VIOLATION)
7.2.0  wave_250: pc=0x7ff734dcbf3c (kernel_code_entry=0x... <...>, kernargs=0x...) (stopped, reason: MEMORY_VIOLATION)

6.3.1  Memory access fault by GPU node-4 (Agent handle: ...) on address ... Reason: ...
7.2.0  OFFLOAD ERROR: memory access fault by GPU 4 (agent ...) at virtual address ... Reasons: ...

The wave regex required pc= and (stopped, reason: to be adjacent; 7.2.0 puts two fields between them. The fault-line lookup was case-sensitive on Memory. Fixed at three sites; the fault line now reuses is_gpu_memory_fault(), which already knows every wording, rather than hardcoding one version's. Both formats are now pinned by fixtures, and neither can be fixed at the other's expense.

This is precisely the failure a single-version fixture cannot catch — it passes while the lane produces nothing. It was found only by running the summarizer against the real file.

Open item 2 (AFAR interaction) — closed, and a number retracted

The agent does not supersede libomptarget. They coexist cleanly:

arm	OFFLOAD ERROR lines	Libomptarget lines
AFAR only	1	8
AFAR + agent	1	8

An earlier report of markers rising 19 → 519 was a grep artifact and is retracted: the extra 500 were agent lines containing __omp_offloading_, matching a case-insensitive search for OFFLOAD. libomptarget's own report is unchanged by the agent.

AFAR OpenMP symbol carries all three facts

__omp_offloading_8116438_1c00689b__QMm_time_steppersPs_tvd_rk_l486

Flang mangling: _QM module m_time_steppers, P procedure s_tvd_rk, _l486. Module, subroutine and line all survive — a different mangling from CCE's OpenACC form, but equally usable.

Volume: the summarizer is mandatory, not nice-to-have

On this lane the agent produces 65,210 lines versus 331 without it — 250 waves rather than 125, roughly 200× the baseline log. Printing that raw on every GPU fault would bury the failure it is meant to explain.

Still open

Item 1 (CCE OpenMP) remains untested — that run is queued. Items 3 (agent overhead, n=1), 4 (summarizer format fragility — now mitigated by two fixtures but not eliminated) and 5 (bench.py / run_case_optimization.sh) stand.

[sbryngelson]

All four lanes measured

CCE OpenMP closes the last cell:

Disassembly for function s_tvd_rk$m_time_steppers_$ck_L486_16:
Memory access fault by GPU node-4 ... Reason: Write access to a read-only page.
125 waves

lane	symbol form	status
NVHPC	Function: s_tvd_rk:438, Line: 486 (free, no agent)	measured
AFAR / OpenMP	__omp_offloading_..._QMm_time_steppersPs_tvd_rk_l486	measured
CCE / OpenACC	s_tvd_rk$m_time_steppers_$ck_L486_6	measured
CCE / OpenMP	s_tvd_rk$m_time_steppers_$ck_L486_16	measured

The symbol form follows the compiler, not the offload model

This corrects the reasoning in the previous update, which treated CCE OpenMP as uncertain because it was a different offload model. It is not: CCE emits its own scheme for both acc and mp, differing only in a trailing counter, while AFAR's Flang form is different again.

Reading any two lanes suggests the offload model picks the mangling. Only running all three shows it is the compiler. All three carry module, subroutine and line, and the summarizer is indifferent to which — but anything that parses these symbols must not assume one scheme per offload model.

Summarizer now validated against three real reports

lane	ROCm	in	out
CCE ACC	6.3.1	14,635	37
CCE OMP	6.3.1	13,826	35
AFAR OMP	7.2.0	65,210	36

Plus a no-agent run still yielding '', so the fallback path is intact.

The stop-PC histogram matters most where it was least expected

CCE OpenMP halts at seven distinct PCs (62/21/19/10/10/2/1), against four for CCE OpenACC and one for AFAR. On that lane, quoting a single PC would be wrong six times in seven — the histogram is not a nicety.

Remaining open

Items 3 (agent overhead, n=1 by decision), 4 (format fragility — two fixtures now, but ROCm formats have already moved once) and 5 (bench.py / run_case_optimization.sh still have no fault handling) stand. Items 1 and 2 are closed.