Derecho site config update for develop; bug fix in setup-meta-modules

[climbfuji]

Note

Added @rickgrubin-noaa/@jim-p-w for the Derecho site config updates, and @AlexanderHrabski-NOAA/@eap for the setup-meta-modules updates.

Description

1. Site config update for Derecho. New environments are based on ncarenv-basic; these require unsetting environment variables (removing entries from PATH and LD_LIBRARY_PATH that ncarenv-basic and ncarenv set and that absolutely shouldn't be in there).
2. Uncovered while working on the Derecho site config: The syntax for Spack's environment modification unset is different than the others: it takes a list of environment variable names, not a dictionary (key : value). This PR fixes it and configures an additional environment modification (remove_path) in spack stack setup-meta-modules.

Dependencies

None

Issues addressed

Never ending Derecho problems

Applications affected

None

Systems affected

Derecho

Testing

• CI: Note whether the automatic tests (GitHub actions tests that run automatically for every commit) pass or not
  • GitHub actions CI tests pass
  • GitHub actions CI tests do not pass (provide explanation)
  • GitHub actions CI tests skipped (provide explanation if necessary)
• New tests added: List and describe any new tests added to GitHub actions
  • ...
• Additional testing: Add information on any additional tests conducted
  • See below

Additional testing on Derecho

jedi-bundle

GCC

Built jedi-bundle with gcc@14.3.0 stack and ran ctests. Had to comment out pyiri-jedi because it complained about a missing fortranformat module, even though the py-fortranformat module was loaded.

Module load instructions

module --force purge

module use /lustre/desc1/scratch/heinzell/spst-derecho-20260717/envs/ue-gcc-14.3.0/modules/Core
module load stack-gcc/14.3.0
module load stack-cray-mpich/8.1.32
module load jedi-fv3-env/1.0.0
module load jedi-mpas-env/1.0.0
module load py-fortranformat

Ctest failures:

Total Test time (real) = 2692.59 sec

The following tests FAILED:
        215 - oops_qg_increment (Failed)                        executable oops
        2114 - ioda_bufr_python_encoder (Failed)                 ioda script
        2115 - ioda_bufr_python_parallel (Failed)                ioda mpi script
        3426 - fv3jedi_staticb_nicas_gfs (Failed)                ci_flake_disable fv3-jedi fv3jedi mpi script
        3428 - fv3jedi_staticb_split_nicas_gfs (Failed)          ci_flake_disable fv3-jedi fv3jedi mpi script
        3429 - fv3jedi_staticb_dirac_local_gfs_6pe (Failed)      ci_flake_disable fv3-jedi fv3jedi mpi script
        3430 - fv3jedi_staticb_dirac_local_gfs_12pe (Failed)     ci_flake_disable fv3-jedi fv3jedi mpi script
        3431 - fv3jedi_staticb_dirac_global_gfs_6pe (Failed)     ci_flake_disable fv3-jedi fv3jedi mpi script
        3432 - fv3jedi_staticb_dirac_global_gfs_12pe (Failed)    ci_flake_disable fv3-jedi fv3jedi mpi script
        3441 - fv3jedi_staticb_cor_geos (Failed)                 ci_flake_disable fv3-jedi fv3jedi mpi script
        3442 - fv3jedi_staticb_nicas_geos (Failed)               ci_flake_disable fv3-jedi fv3jedi mpi script
        3460 - fv3jedi_hyb-3dvar_gfs (Failed)                    ci_flake_disable fv3-jedi fv3jedi mpi script

Intel oneAPI

Built jedi-bundle with oneapi@2025.3.2 stack and ran ctests. Had to comment out pyiri-jedi because it complained about a missing fortranformat module, even though the py-fortranformat module was loaded.

Module load instructions

module --force purge

module use /lustre/desc1/scratch/heinzell/spst-derecho-20260717/envs/ue-oneapi-2025.3.2/modules/Core
module load stack-intel-oneapi-compilers/2025.3.2
module load stack-cray-mpich/8.1.32
module load jedi-fv3-env/1.0.0
module load jedi-mpas-env/1.0.0
module load py-fortranformat

Ctest failures:

Total Test time (real) = 5933.99 sec

The following tests FAILED:
        2112 - ioda_bufr_python_encoder (Failed)                 ioda script
        2113 - ioda_bufr_python_parallel (Failed)                ioda mpi script
        2706 - ufo_gnssrobendmetoffice_qc (Timeout)              ci_oneapi_disable filters mpi script ufo
        2708 - ufo_gnssrobendmetoffice_obserror (Timeout)        ci_oneapi_disable filters mpi script ufo
        2714 - ufo_gnssro_super_refraction_check (Timeout)       ci_oneapi_disable filters mpi script ufo
        2982 - ufo_opr_gnssrobendmetoffice (Timeout)             ci_oneapi_disable mpi operators script ufo
        2983 - ufo_linopr_gnssrobendmetoffice (Timeout)          ci_oneapi_disable mpi operators script ufo
        2984 - ufo_opr_gnssrobendmetoffice_nopseudo (Timeout)    ci_oneapi_disable mpi operators script ufo
        2985 - ufo_linopr_gnssrobendmetoffice_nopseudo (Timeout) ci_oneapi_disable mpi operators script ufo
        2987 - ufo_opr_gnssrobendmetoffice_nosupercheck (Timeout) ci_oneapi_disable mpi operators script ufo
        2988 - ufo_opr_gnssrobendmetoffice_nosupercheck_missingdata (Timeout) ci_oneapi_disable mpi operators script ufo
        2989 - ufo_linopr_gnssrobendmetoffice_nosupercheck (Timeout) ci_oneapi_disable mpi operators script ufo
        2990 - ufo_opr_gnssrobendmetoffice_refractivity_changes (Timeout) ci_oneapi_disable mpi operators script ufo
        2991 - ufo_linopr_gnssrobendmetoffice_refractivity_changes (Timeout) ci_oneapi_disable mpi operators script ufo
        3124 - ufo_opr_gmi_scan_angles (Timeout)                 crtm mpi operators script tier2 ufo
        3125 - ufo_linopr_gmi_scan_angles (Timeout)              crtm mpi operators script tier2 ufo
        3424 - fv3jedi_staticb_nicas_gfs (Failed)                ci_flake_disable fv3-jedi fv3jedi mpi script
        3426 - fv3jedi_staticb_split_nicas_gfs (Failed)          ci_flake_disable fv3-jedi fv3jedi mpi script
        3427 - fv3jedi_staticb_dirac_local_gfs_6pe (Failed)      ci_flake_disable fv3-jedi fv3jedi mpi script
        3428 - fv3jedi_staticb_dirac_local_gfs_12pe (Failed)     ci_flake_disable fv3-jedi fv3jedi mpi script
        3429 - fv3jedi_staticb_dirac_global_gfs_6pe (Failed)     ci_flake_disable fv3-jedi fv3jedi mpi script
        3430 - fv3jedi_staticb_dirac_global_gfs_12pe (Failed)    ci_flake_disable fv3-jedi fv3jedi mpi script
        3439 - fv3jedi_staticb_cor_geos (Failed)                 ci_flake_disable fv3-jedi fv3jedi mpi script
        3440 - fv3jedi_staticb_nicas_geos (Failed)               ci_flake_disable fv3-jedi fv3jedi mpi script
        3458 - fv3jedi_hyb-3dvar_gfs (Failed)                    ci_flake_disable fv3-jedi fv3jedi mpi script
        3558 - mpasjedi_hofx3d (Timeout)                         ci_oneapi_disable mpasjedi mpi script

UFS Weather Model

See https://github.com/climbfuji/ufs-weather-model/tree/feature/ss2.x_dom_derecho

This branch originally started from ufs-community/ufs-weather-model#3174, but I updated it to work with the current package versions from spack-stack develop.

With GNU and Intel oneAPI, I was able to run control_c48. I tried a few other tests and was surprised to find how fragile these are. Some tests (32bit_sfs_debug) failed with out-of-bounds errors at runtime, others with errors related to too long names of the Fortran objects in the auto-generated CCPP caps - how comes these tests pass elsewhere? Or is it because my branch started from ufs-community/ufs-weather-model#3174?

Checklist

• This PR addresses one issue/problem/enhancement or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.
• All necessary updates to the documentation (spack-stack wiki) will be made when this PR is merged

[climbfuji]

Probably not needed

[rickgrubin-noaa]

Questions to help my clarity of the situation:

• using the sytem-provided oneAPI compiler version rather than the EPIC-installed version
  • better to use the system-provided version for "better interoperability" with MPI, etc., or ?
• using the GCC compilers @ 14.3.0
  • package versions for release/2.1 all build? I vaguely (incorrectly?) recall that v14.x was a no-go for some packages

and

Uncovered while working on the Derecho site config: The syntax for Spack's environment modification unset is different than the others

where "others" is the shell, or ?

[climbfuji]

Questions to help my clarity of the situation:

• using the sytem-provided oneAPI compiler version rather than the EPIC-installed version

  • better to use the system-provided version for "better interoperability" with MPI, etc., or ?

• using the GCC compilers @ 14.3.0

  • package versions for release/2.1 all build? I vaguely (incorrectly?) recall that v14.x was a no-go for some packages

and

Uncovered while working on the Derecho site config: The syntax for Spack's environment modification unset is different than the others

where "others" is the shell, or ?

This pull request is for develop. release/2.1 is not addressed here.

The Derecho ncarenv doesn't provide a GCC 13 module, only 14 or 12.

The system-provided Intel oneAPI compiler means avoiding one manual MODULEPATH, and it's slightly newer (2025.3.2 instead of 2025.3.1).

unset takes a list of variable names. All others (remove_path, prepend_path, set, ...) take key : value pairs in the Spack yamls.

[rickgrubin-noaa]

This pull request is for develop. release/2.1 is not addressed here.

Thanks for clarifying.

I'll let EPIC folks know that issue Derecho-specific issues for the WM spack-stack v2.1.x upgrade #3306 and adjacent PR Upgrade WM to spack-stack 2.1.x- #3174 are OBE.

unset takes a list of variable names. All others (remove_path, prepend_path, set, ...) take key : value pairs in the Spack yamls.

So a spack-specific command; thanks.

[eap]

I didn't test this manually, but I assume you have. I read through the meta-module logic and the new extra modules, the changes look sensible to me.

[climbfuji]

@rickgrubin-noaa @jim-p-w This PR is ready. I can't spend more time on Derecho and UFS than that.