Skip to content

DEMdemo_PlateSinkage: collect forces in the force kernel, and documen… - #80

Open
DanNegrut wants to merge 1 commit into
mainfrom
plate-sinkage-inkernel-collection
Open

DEMdemo_PlateSinkage: collect forces in the force kernel, and documen…#80
DanNegrut wants to merge 1 commit into
mainfrom
plate-sinkage-inkernel-collection

Conversation

@DanNegrut

Copy link
Copy Markdown
Collaborator

…t the measured gain

By default DEME accumulates contact forces onto their owners in a separate pass over the contact array (forceToAcc) after the force kernel has run. SetCollectAccRightAfterForceCalc folds that accumulation into the force kernel itself. Its API comment said this "may give some performance boost if you have only polydisperse spheres, no clumps"; measured on this demo, which uses 3-sphere clumps, it is the largest single performance lever in the code, so the demo now turns it on and the comment now states what was measured.

Method: identical source on both machines, fixed step 5e-6 s (about 310,000 steps at every size), one GPU each, throughput taken over the physics loops only so JIT compilation is excluded. Three random bed packings per cell, eleven for the two 152,670-grain default cells. Million grain-steps per second:

grains    Blackwell default  Blackwell in-kernel  MI350X default  MI350X in-kernel
152,670        142.1            171.4 (+21%)          142.0          278.8  (+96%)
308,486        134.5            166.3 (+24%)          137.1          335.5 (+145%)
613,998        128.5            155.0 (+21%)          124.6          378.3 (+204%)

Blackwell is an RTX PRO 6000 workstation card under CUDA; MI350X is under ROCm 7.2 via the HIP build. On the default path the two tie within 2%; with in-kernel collection the MI350X leads by 1.6x to 2.4x. The reason is visible in the per-phase timers: the separate forceToAcc pass is 18% of dT time on CUDA and 83% on HIP, and on HIP it also slows the concurrent kT phases (binning, pair search, history map) by 2.5x to 2.9x, presumably through memory-system contention, since kT's code is untouched by this option.

The other alternative, UseCubForceCollection, was measured too and is slower than the default on both backends (-55% to -59% on CUDA at every size; -26% to +69% on HIP), so in-kernel collection is the right choice here rather than one of three.

Physics is essentially unchanged. Plate pressure at 10 mm sinkage shifts between the two collection modes by 0.8% to 6.0% across the six machine-and-size cells, against a bed-to-bed scatter of 4% to 19% from reseeding alone, so the shift is at or below the test's own repeatability. In-kernel reads slightly higher in five of the six cells, which is what a different atomic accumulation order rounding differently would look like.

What in-kernel collection gives up: tracker force-pair queries throw while it is on (DEMTracker::assertThereIsForcePairs). This demo reads the plate load through ContactAcc, which goes via GetOwnerAcc and is not guarded, and contact-force recording and output are unaffected in general; the comment now says so, since the previous text did not mention it.

Whether the library default should change is left to the maintainer; this commit only changes the demo and the documentation.

…t the measured gain

By default DEME accumulates contact forces onto their owners in a separate pass over
the contact array (forceToAcc) after the force kernel has run. SetCollectAccRightAfterForceCalc
folds that accumulation into the force kernel itself. Its API comment said this "may give
some performance boost if you have only polydisperse spheres, no clumps"; measured on this
demo, which uses 3-sphere clumps, it is the largest single performance lever in the code,
so the demo now turns it on and the comment now states what was measured.

Method: identical source on both machines, fixed step 5e-6 s (about 310,000 steps at
every size), one GPU each, throughput taken over the physics loops only so JIT compilation
is excluded. Three random bed packings per cell, eleven for the two 152,670-grain default
cells. Million grain-steps per second:

    grains    Blackwell default  Blackwell in-kernel  MI350X default  MI350X in-kernel
    152,670        142.1            171.4 (+21%)          142.0          278.8  (+96%)
    308,486        134.5            166.3 (+24%)          137.1          335.5 (+145%)
    613,998        128.5            155.0 (+21%)          124.6          378.3 (+204%)

Blackwell is an RTX PRO 6000 workstation card under CUDA; MI350X is under ROCm 7.2 via
the HIP build. On the default path the two tie within 2%; with in-kernel collection the
MI350X leads by 1.6x to 2.4x. The reason is visible in the per-phase timers: the separate
forceToAcc pass is 18% of dT time on CUDA and 83% on HIP, and on HIP it also slows the
concurrent kT phases (binning, pair search, history map) by 2.5x to 2.9x, presumably
through memory-system contention, since kT's code is untouched by this option.

The other alternative, UseCubForceCollection, was measured too and is slower than the
default on both backends (-55% to -59% on CUDA at every size; -26% to +69% on HIP), so
in-kernel collection is the right choice here rather than one of three.

Physics is essentially unchanged. Plate pressure at 10 mm sinkage shifts between the two
collection modes by 0.8% to 6.0% across the six machine-and-size cells, against a
bed-to-bed scatter of 4% to 19% from reseeding alone, so the shift is at or below the
test's own repeatability. In-kernel reads slightly higher in five of the six cells, which
is what a different atomic accumulation order rounding differently would look like.

What in-kernel collection gives up: tracker force-pair queries throw while it is on
(DEMTracker::assertThereIsForcePairs). This demo reads the plate load through
ContactAcc, which goes via GetOwnerAcc and is not guarded, and contact-force recording
and output are unaffected in general; the comment now says so, since the previous text
did not mention it.

Whether the library default should change is left to the maintainer; this commit only
changes the demo and the documentation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant