Error on MPI+CUDA, compatibility with Newton-Krylov, fix OpenMP deadlock in Newton-Krylov - #2887
Merged
Conversation
…e GPU Three CUDA combinations that silently produced wrong results now fail at config time instead: - More than one MPI rank. The solver keeps its work vectors on the device for the whole Krylov iteration (CSysSolve::UploadSystem) while the halo exchange is host-side, so a partitioned run converges to the wrong answer. - The AD and direct differentiation solvers. nvcc cannot compile the CoDiPack types, so the kernels are only linked into the primal libraries (SU2_ENABLE_CUDA_KERNELS). This used to fail minutes into the run with a message claiming CUDA had not been compiled in, which is misleading for a build configured with -Denable-cuda=true; GPUNotAvailable now tells the two cases apart as well. Newton-Krylov was the third, and is fixed rather than rejected. Its outer matrix-vector product is matrix free and host resident, but the preconditioner is an ordinary CSysMatrix operation that the device can do. It went through neither CSysSolve::Solve (so the vectors were never uploaded) nor CSysMatrixVectorProduct (so the Jacobian was never uploaded), and every linear solve exited immediately having done nothing, with all preconditioners giving identical results. The Jacobian is now uploaded before Build(), and ApplyPreconditionerOnDevice (the mirror of the existing ApplyPreconditionerOnHost) transfers the vectors around the apply. Device expressions are enabled for the duration so a nested Krylov solve offloads too: verified with Tracy that all of its matrix-vector products and inner preconditioner applies run on the device, for the nested FGMRES, BCGSTAB and SMOOTHER inner solvers. On ONERA M6 RANS the GPU results track the CPU ones to 5-6 significant figures and are 1.3-1.9x faster per iteration. Also drops the SU2_OMP_MASTER around the force calculations, which #2870 parallelized with OpenMP. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Changes
All in the title.
PR Checklist
pre-commit run --allto format old commits.