Parallelize release system tests via per-tutorial matrix

[PranjalManhgaye]

Summary

Parallel release system tests via a per-tutorial GitHub Actions matrix derived from tests.yaml.

Approach: A prepare-matrix job reads release, groups cases by tutorial, and emits one job per tutorial with comma-separated per-case suite names (not bare tutorial suite names, so cases outside release are not pulled in).

Example: aste-turbine => 1 case, elastic-tube-1d => 6 cases. All 40 jobs together cover full release (67 cases), no duplicates.

Manual workflow:

• release => serial by default (single job)
• release + parallel: true => per-tutorial matrix (~40 jobs)
• any other suites => single job (unchanged)

Nightly: still runs release as one serial job (unchanged).

Closes #789. Replaces the earlier shard-based approach in this PR.

Test plan

• prepare-matrix job on ubuntu-latest
• Manual release serial (default)
• Manual release with parallel: true on self-hosted VM (#33298363903)

[PranjalManhgaye]

@MakisH follow-ups (i think later, not this PR) :

• same matrix for manual / latest-components workflows
• more shards if we get more runners
• docker build sharing only if ci shows we need it
• option 2 (systemtests.py parallel) => skipped for now

[MakisH]

Nice to see a first prototype towards parallization!

While this is a valid and easy-to-implement approach for parallelization, I think it mainly adds a layer to the current approach.

Ideally, we should end up with the individual test suites (the ones per tutorial) as shards, so that the jobs also get meaningful names. Issues there will be:

• race conditions in building the same Docker layers, if a runner picks more than one shard at the same time,
• that some too long tests are currently excluded from the release_test (see extra), as these take too long. We could then just define these directly in the extra, instead of referring to the ones defined at the tutorial level with anchors.

Nevertheless, I could create another runner and use this PR to test if the parallelism with multiple custom runners makes sense.

[MakisH]

@PranjalManhgaye #842 introduced some conflicts in system-tests-latest-components.yml that should be easy to resolve.

[MakisH]

I triggered a test run: https://github.com/precice/tutorials/actions/runs/27600082238

[MakisH]

While the matrix is a good approach, I think we should drop the shards and instead reuse the per-tutorial test suites (elastic-tube-1d, elastic-tube-3d, ...). This should also give clearer and faster output, and these test suites are by definition not overlapping with each other.

What I am not sure at the moment is what is the right way to get these inputs to avoid duplication.

[PranjalManhgaye]

@MakisH what do you think ; add a small prepare-matrix job in the workflow that reads tests.yaml , or should we extend systemtests.py tooling to expose this list somehow ??

[MakisH]

Maybe related: I always have to count how many tutorials and tutorial cases we have, both for reporting and now for ensuring that everything is tested. We could collect this information from all the metadata.yaml (assuming that they are complete, or somehow checking with another script if these are complete, checking all available directories).

We could then use that information as input for this matrix.

But in general, a prepare-matrix job would work. Note, however, that tests.yaml includes some duplicate information. We could define the release as all tutorials and read that. The question is then what do we do with the extra that take longer to run.

[PranjalManhgaye]

@MakisH i am happy to follow this and drop the artificial shards , and i can implement it by adding a small prepare-matrix step that reads tests.yaml so the workflow doesn’t duplicate suite lists, but to confirm before i do, do you want the matrix at tutorial-suite level (not per-case), keep release as the source of truth there for now, and keep extra out of the default latest-components run (manual or less frequent) ??

[MakisH]

Overall, I would say that the whole feature is still in the exploration phase and would need some research / brainstorming.

do you want the matrix at tutorial-suite level (not per-case)

We could also split per-case, as long as we don't run a case multiple times. That would give even more refined output. But I think it would be simpler to split per-tutorial, which would also give simpler names. It would also give fewer jobs, which might be easier to manage for the runners, and will share some of the setup overhead.

keep release as the source of truth there for now, and keep extra out of the default latest-components run (manual or less frequent) ??

Yes, I would keep this as-is for now, unless it makes sense to change it.

[MakisH]

@PranjalManhgaye could you please describe (in a comment or in the PR description) in more detail your current approach?

Testing in https://github.com/precice/tutorials/actions/runs/33246850254

Some first observations:

• Each parallel step runs all the available case combinations of the release test suite.
• Each run takes longer due to the setup and teardown. If we add more runners, the total time can be significantly reduced.
• The build artifacts are small (but in total larger than in the case of the serial run, as we duplicate the tools/ and tests/).
• The total job summary is getting a bit long due to duplicated information, but if we ran a test suite in serial, this information would be useful. GitHub already hides subsequent summaries by default and asks to explicitly load them.

Issues:

• Every job appears on the side as run_system_tests, making it difficult to get an overview. Could we get the tutorial name there, instead?
• It is not clear from the beginning how many jobs will be executed. The number of matrix entries grows continuously. Suggestions:
  • In the Parallel test matrix job summary, explain what happens next.
  • In the list of tests, add numbers, to get an idea of the progress of the run.
  • Why do we need to limit the concurrent jobs? In any case, they will be limited by the number of available runners.
• All build artifacts have the same name. We should probably add the job id to the name.

Questions:

• What happens if I run another test suite?
• Can I still run the complete release test suite in serial?

I have not looked deeper; I will do that once the overview is clear.

To make this easier to merge, I would recommend making the parallel run optional, and keeping the serial by default.

[PranjalManhgaye]

Thanks @MakisH for running this on the VM.

• Each matrix job runs only that tutorial’s release cases (comma-separated per-case suite names from tests.yaml), not the full 67-case release suite, for example aste-turbine runs 1 case, elastic-tube-1d runs 6.

• Other suites: manual workflow with suites != release still goes through the single system-tests.yml job unchanged.

• Serial release: currently nightly + manual release always use the matrix. I’ll add an option to keep serial as default and make parallel opt-in, as you suggested.

• I’ll also address: $/ workflow syntax (#901), job/artifact naming, prepare-matrix summary, and rebase onto develop.

[PranjalManhgaye]

I guess @MakisH we need to re-run to check serial as default and parallel as option.

[MakisH]

Test runs:

• quickstart in serial: https://github.com/precice/tutorials/actions/runs/33297820153
  • It looks a bit strange to have the unused run_system_tests job underneath the one that ran.
• quickstart with ticking the parallel matrix box: runs the same as in serial: https://github.com/precice/tutorials/actions/runs/33297957480
• quickstart,flow-over-heated-plate_fluid-openfoam_solid-openfoam with ticking the parallel matrix job: runs the same as in serial: https://github.com/precice/tutorials/actions/runs/33298007316
• release without the checkbox: runs as usual, in serial: https://github.com/precice/tutorials/actions/runs/33298084758 (cancelled)
• release with the checkbox: runs in parallel, as discussed https://github.com/precice/tutorials/actions/runs/33298363903

I will return to the individual points later. The job names and logs/archive names are already clearer.

[PranjalManhgaye]

@MakisH Thanks for testing,

• Parallel checkbox only applies to exact release; quickstart staying serial is correct.
• Extra grey run_system_tests jobs are skipped reusable-workflow placeholders in the UI, not duplicate runs.
• Parallel run failures were artifact ETIMEDOUT, not test failures - we can re-run those jobs or disable _full uploads for matrix runs.

and yes ofc that strange thing in ui is coz of the manual workflow has 3 conditional jobs that all call the same reusable workflow - only the one that actually runs does real work ; I guess the possible fix could be : merge serial + custom into one job (2 jobs total) so fewer skipped branches appear in the sidebar, what do you think abt it ?

ready to tweak anything else when you revisit the remaining points.

[MakisH]

* Parallel run failures were artifact `ETIMEDOUT`, not test failures - we can re-run those jobs or `disable _full uploads` for matrix runs.

Starting the failing jobs again, the issue does not seem to be persistent: https://github.com/precice/tutorials/actions/runs/33298363903 Still, one reason to keep the parallel matrix optional.

I am not sure if this or making a separate workflow would be the best option. I lean towards the separate workflow, but I need to have a closer look.