SRE-3893 build: Consolidate build scripts by grom72 · Pull Request #19016 · daos-stack/daos

grom72 · 2026-09-04T11:11:37Z

Summary

Consolidate dependency installation, dependency builds, DAOS builds,
package generation, and RPM verification under utils/build. Use these
entry points consistently from Jenkins, Docker builds, and bare-host
workflows while retaining support for custom scons options.

By default, generate dependency and DAOS RPMs directly under
<repo_root>/rpms/deps and <repo_root>/rpms/daos, respectively, and
create repository metadata under <repo_root>/rpms/repodata. Allow
RPM_OUTPUT_DIR to override the repository root.

Add distro auto-detection, optional external dependency repositories,
and configurable package-verification severity. Remove the superseded
wrappers from ci/rpm and utils/rpms.

Update Jenkins to copy dependency RPMs produced in the Docker build
image into the workspace repository so that dependency and DAOS RPMs
are included in the artifact upload.

Changes

Add shared utils/build scripts for:
- installing pre-built dependencies;
- building missing dependencies;
- building DAOS;
- generating and verifying packages.
Remove the superseded ci/rpm and utils/rpms/build_packages.sh wrappers.
Add automatic EL and openSUSE Leap distro detection.
Support an optional temporary dependency repository through
DAOS_DEPS_EXT_REPO for the install_deps.sh script (to be used mainly as bare-host builds pre-step).
Allow additional options and variables to pass through to scons in build_daos.sh and build_deps.sh scripts.
Update the EL9 and Leap 15 Docker build flows to use the unified scripts.
Update Jenkins RPM generation and artifact collection.
Document the unified four-step build procedure.

Package verification

Package verification runs automatically after each nonempty RPM group.

verify=yes treats validation findings as errors.
verify=no reports validation findings as warnings.
Missing tooling and other critical setup errors remain fatal.

Requires:

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-09-04T11:11:57Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3893

daosbuild3 · 2026-09-04T11:33:42Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/2/execution/node/442/log

daosbuild3 · 2026-09-04T13:27:26Z

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/3/execution/node/493/log

daosbuild3 · 2026-09-07T09:27:31Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/10/execution/node/858/log

Consolidate dependency installation, dependency builds, DAOS builds, package generation, and RPM verification under utils/build. Use these entry points consistently from Jenkins, Docker builds, and bare-host workflows while retaining support for custom scons options. By default, generate dependency and DAOS RPMs directly under <repo_root>/rpms/deps and <repo_root>/rpms/daos, respectively, and create repository metadata under <repo_root>/rpms/repodata. Allow RPM_OUTPUT_DIR to override the repository root. Add distro auto-detection, optional external dependency repositories, and configurable package-verification severity. Remove the superseded wrappers from ci/rpm and utils/rpms. Update Jenkins to copy dependency RPMs produced in the Docker build image into the workspace repository so that dependency and DAOS RPMs are included in the artifact upload. skip-cancel-previous-builds: true skip-python-bandit-check: true skip-unit-tests: true skip-test: true skip-test-hardware: true skip-functional-on-leap-15: false Priority: 2 Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Document the shared scripts for installing dependencies, building DAOS, generating packages, and creating a complete RPM repository. Update Docker examples and packaging script paths to match the new layout. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> skip-cancel-previous-builds: true skip-python-bandit-check: true skip-unit-tests: true skip-test: true skip-test-hardware: true skip-functional-on-leap-15: false

Move distro detection from build_packages.sh to build_utils.sh so it can also be reused by install_deps.sh. Propagate detection failures explicitly from command substitutions and avoid importing variables from /etc/os-release into the calling shell. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2

janekmi · 2026-09-10T23:13:02Z

+ARG DAOS_TARGET_TYPE=
+ARG DEPS_JOBS=
+
+ARG DAOS_BUILD_TYPE=$DAOS_TARGET_TYPE


Why these arguments are named differently by mean the same thing? 😕

Please ask scons ;)

janekmi · 2026-09-10T23:38:54Z

+        utils/build/build_deps.sh \
+            ${DEPS_JOBS:+--jobs "$DEPS_JOBS"} \
+            ${DAOS_BUILD_TYPE:+BUILD_TYPE="$DAOS_BUILD_TYPE"} \
+            ${DAOS_TARGET_TYPE:+TARGET_TYPE="$DAOS_TARGET_TYPE"} && \


Does truely the cleanup step depend on the build step hence you need && ?

It is not about cleanup it is about failing the whole Docker step.

janekmi · 2026-09-10T23:40:53Z

+        [ "$DAOS_KEEP_BUILD" != "no" ] || { \
        rm -rf .sconf_temp .sconsign.dblite config.log && \
-        ([ "$DAOS_KEEP_BUILD" != "no" ] || /bin/rm -rf build *.gz); \
+            /bin/rm -rf build *.gz; \


Indentation.

/bin/rm seriously?

You can probably combine both calls.

Suggested change

/bin/rm -rf build *.gz; \

rm -rf .sconf_temp .sconsign.dblite config.log build *.gz;

Done
4aca75c

janekmi · 2026-09-10T23:49:23Z

 ARG DAOS_DEPS_BUILD
 ARG DAOS_BUILD=$DAOS_DEPS_BUILD
-ARG DAOS_JAVA_BUILD=$DAOS_BUILD
+ARG DAOS_JAVA_BUILD=no


Redundant?

Suggested change

ARG DAOS_JAVA_BUILD=no

ARG DAOS_JAVA_BUILD

Good catch ;)
4aca75c

janekmi · 2026-09-10T23:52:58Z

+        mkdir -p /home/daos/daos/src && \
+       cp -r /tmp/daos-src/. /home/daos/daos/src && \


Indentation.

Suggested change

mkdir -p /home/daos/daos/src && \

cp -r /tmp/daos-src/. /home/daos/daos/src && \

mkdir -p /home/daos/daos/src && \

cp -r /tmp/daos-src/. /home/daos/daos/src && \

daosbuild3 · 2026-09-11T07:30:08Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/36/execution/node/1869/log

Replace positional build and verification modes with named options: --build-range, --rpm-suffix, -Werror, and -Wno-error. Make the package output directory optional and update all Jenkins and Docker callers accordingly. Update dependency and verification script usage documentation, support command-line BUILD_TYPE cleanup with --force, fix Docker ownership and RPM output path handling, and synchronize the development guide with the new interfaces. Address review comments. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Doc-only: true Priority: 2 skip-cancel-previous-builds: true

daosbuild3 · 2026-09-14T15:01:53Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19016/37/execution/node/1745/log

knard38

Mostly looks good to me.
Some points to clarify and/or fix.

knard38 · 2026-09-15T06:42:58Z

+        BUILD_TYPE=*)
+            build_type="${arg#BUILD_TYPE=}"
+            ;;


From my understanding scons prefers TARGET_TYPE over BUILD_TYPE for naming the real prereq directory whenever TARGET_TYPE is set to anything other than default (site_scons/prereq_tools/base.py's _setup_build_type()), and this script has no TARGET_TYPE=* case at all.

If I am correct, this is what it should looks like in practice:

$ utils/build/build_deps.sh --force TARGET_TYPE=debug Removing "/opt/daos/prereq/release" # untouched artifacts, wiped anyway Removing "build/external/release" # same ... # scons then builds with TARGET_TYPE=debug, i.e. into # prereq/debug/ -- the one directory --force never cleaned

So --force wipes the wrong (release) tree and leaves the one that's actually about to be (re)built (debug) alone -- the opposite of what -f/--force promises.

A possible fix could be to mirroring scons's own precedence (prefer TARGET_TYPE when given, else fall back to BUILD_TYPE):

target_type="" build_type=release for arg in "$@"; do case "$arg" in ... BUILD_TYPE=*) build_type="${arg#BUILD_TYPE=}" ;; TARGET_TYPE=*) target_type="${arg#TARGET_TYPE=}" ;; esac done ttype="${target_type:-$build_type}" # ... then use "${ttype}" instead of "${build_type}" in # both rm -rf lines below

good-catch :) c88d6fe

knard38 · 2026-09-15T07:07:13Z

+      ;;
+  esac
+done
+verify_mode="${verify_mode:--Werror}"


From my understanding, the Jenkins bare-host "Build DAOS RPMs" step calls with no -Werror/-Wno-error flag (line 853 and 919 of the Jenkinsfile), so it runs with the script's own default (-Werror, strict).

Meanwhile both Docker-image RPM-build steps explicitly do:

set -- --build-range=daos /home/daos/rpms; [ "$DAOS_PACKAGES_VERIFY" = "yes" ] || set -- "$@" -Wno-error;

Thus, they stay lenient by default unless DAOS_PACKAGES_VERIFY=yes is explicitly set. From my investigation, I was not able to find usage of this last variable. If I am correct, the rpm used by the Jenkins bare host and the one from the docker image are not consistent.
Should the bare-host call also pass -Wno-error for parity, or is stricter-by-default intentional here?

The goal is to let Dockerfile user to disable RPMs validation.
The default behavior is to verify RPMs and the way to disable it (but still get warnings is something is wrong) is to use --build-arg DAOS_PACKAGES_VERIFY=no

knard38 · 2026-09-15T07:17:37Z

+
+
+# The script can be used only on el9 and leap/sles 15
+validate_rpm_suffix() {


Could we put this function in build_utils.sh and replace equivalent inline code in verify_packages.sh and build_packages.sh ?

Do You mean install_deps.sh?

The build_package.sh script can also be used for Debian/Ubuntu:
OUTPUT_TYPE=deb ./utils/build/build_packages.sh --build-range=deps

validate_rpm_suffix() has been moved to build_utils.sh -> c88d6fe

knard38 · 2026-09-15T08:20:31Z

+    case "$arg" in
+        -c)
+            scons -c
+            exit 0
+            ;;


Suggested change

case "$arg" in

-c)

scons -c

exit 0

;;

-c)

scons -c

rm -rf .sconf_temp .sconsign.dblite config.log build .build_vars.sh .build_vars.json

exit 0

;;

--full-clean)

scons -c

rm -rf .sconf_temp .sconsign.dblite config.log build .build_vars.sh .build_vars.json daos.conf

exit 0

;;

From my experience, the scons -c is not fully reliable. To have a proper cleanup, I have to remove some files generated by the build process. Most of the time, I am doing a full cleanup with removing the daos.conf file generated: without this it reuse the initial setting even if I redefine it on the command line.

However, I am not sure of the exact goal behind the -c that you aim ?

Fxed: c88d6fe

knard38 · 2026-09-15T08:33:05Z

   make sure the file is referenced by the
   `RPM_CHANGELOG="<component>.changelog"` variable in `utils/rpms/<component>.sh`.
+
+## Unified DAOS Build Procedure


From my understanding, most of these new scripts are not yet supporting Debian like distribution.
It should be valuable, to clearly to define the status of the support of such distribution.
From my side, I am still able to build Debian package as before and thus I have not detected regression.

Documentation updated: c88d6fe

- Align build and dependency script usage messages. - Handle help and clean options directly in each script. - Validate supported RPM suffixes through shared helpers. - Use SCons defaults for build_daos dependency handling. - Improve dependency cleanup for BUILD_TYPE and TARGET_TYPE. - Reuse build_daos cleanup in the EL9 and Leap Dockerfiles. - Clarify supported distributions and build-script behavior in the docs. - improving the help content - usage() Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

- Add full-clean support to build_daos.sh while preserving build metadata for the regular clean path. - Keep Docker dependency-stage cleanup independent of the full DAOS source tree and preserve build variables required for RPM packaging. - Suppress unnecessary RPM packaging xtrace output. - Improve usage formatting for build and dependency scripts. - Clarify RPM suffix, dependency installation, verification, and cleanup documentation. - Make read-only site_scons cache cleanup non-fatal. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 force-pushed the grom72/SRE-3893-script2build branch from f31fa70 to db5ee07 Compare September 4, 2026 11:21

grom72 force-pushed the grom72/SRE-3893-script2build branch 4 times, most recently from a6ee981 to b423038 Compare September 4, 2026 20:37

grom72 force-pushed the grom72/SRE-3893-script2build branch 7 times, most recently from e0cbb87 to 32eb1f3 Compare September 9, 2026 15:03

janekmi requested changes Sep 9, 2026

View reviewed changes

grom72 force-pushed the grom72/SRE-3893-script2build branch 4 times, most recently from 6deddc7 to 93864f9 Compare September 10, 2026 09:23

grom72 added 3 commits September 10, 2026 18:20

grom72 force-pushed the grom72/SRE-3893-script2build branch from 93864f9 to 9d48806 Compare September 10, 2026 16:21

Final documentation polishing

282c33c

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2

grom72 requested a review from janekmi September 10, 2026 16:53

grom72 changed the title ~~SRE-3893 build: script2build~~ SRE-3893 build: Consolidate build scripts Sep 10, 2026

grom72 marked this pull request as ready for review September 10, 2026 16:54

grom72 requested a review from a team as a code owner September 10, 2026 16:54

grom72 requested review from daltonbohning, knard38 and phender and removed request for a team September 10, 2026 16:54

janekmi requested changes Sep 10, 2026

View reviewed changes

grom72 requested a review from janekmi September 14, 2026 08:29

Fix script documentation

0e0e9f7

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Doc-only: true Priority: 2 skip-cancel-previous-builds: true

knard38 reviewed Sep 15, 2026

View reviewed changes

grom72 requested a review from knard38 September 15, 2026 21:49

	/bin/rm -rf build *.gz; \
	rm -rf .sconf_temp .sconsign.dblite config.log build *.gz;

		mkdir -p /home/daos/daos/src && \
		cp -r /tmp/daos-src/. /home/daos/daos/src && \



		# The script can be used only on el9 and leap/sles 15
		validate_rpm_suffix() {

Uh oh!

Conversation

grom72 commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Package verification

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 4, 2026

Uh oh!

daosbuild3 commented Sep 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Sep 11, 2026

Uh oh!

daosbuild3 commented Sep 14, 2026

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grom72 Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grom72 Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grom72 Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

grom72 commented Sep 4, 2026 •

edited

Loading

grom72 Sep 15, 2026 •

edited

Loading

grom72 Sep 15, 2026 •

edited

Loading

grom72 Sep 15, 2026 •

edited

Loading

grom72 Sep 15, 2026 •

edited

Loading