build(test): stop test forks oversubscribing the machine - #16158
Conversation
The `tasks.named('isolatedTestsTwo', Test)` block set `maxParallelForks = 1`
and `forkEvery = 100`, but it was registered BEFORE the
`tasks.withType(Test).configureEach` block in the same script. Gradle runs
both as deferred configuration actions in registration order at task
realization, so the later `configureEach` overwrote both values and the task
ran with `configuredTestParallel` forks instead of one.
Its three test patterns have been deliberately serialized since 2013 because
they are order sensitive, so running them in parallel risked exactly the kind
of static-state flakiness the suite is isolated to avoid.
Move the override after the `configureEach` block so it wins, and add a
comment recording the ordering requirement.
CI impact is negligible: the task filters three classes and sharding assigns
the whole task to a single shard, so `forkEvery = 100` is never reached.
Assisted-by: claude-code:claude-opus-5
A plain `./gradlew build` could leave a developer's workstation unusable. Three multipliers stacked, each reasonable on its own: 1. `availableProcessors()` reports LOGICAL processors - SMT threads on x64, and on Apple silicon every efficiency core as well as every performance core. Taking 3/4 of that already overstates real capacity. 2. `maxParallelForks` is per Test task, and with `org.gradle.parallel=true` several test-bearing modules run at once, so the real ceiling is the worker-lease pool rather than any single task's fork count. 3. Every forked JVM sizes its own GC and JIT thread pools for the WHOLE machine, because no fork knows the others exist. On a 20-processor host that is 15 ParallelGCThreads + 4 ConcGCThreads + 12 CICompilerCount = 31 threads per fork before a single test runs. Gradle cannot see the third one: it charges each fork a single worker lease, as though a fork were one thread. Gradle's own default for maxParallelForks is 1 for exactly that reason; raising it opts out of that protection. Two changes, applied to all three builds in this repository (root, grails-gradle and grails-forge each have their own settings.gradle): - Local test forks drop from 3/4 of the logical processors to half. CI keeps its existing budget, so CI fork counts are unchanged. - Every test fork is told how many processors it may size its thread pools from, as availableProcessors / maxWorkerCount. The denominator is the BUILD-WIDE worker limit rather than any one task's maxParallelForks, because that is what actually bounds how many forks run concurrently. A floor of 2 keeps G1 rather than dropping to Serial GC. It is supplied through jvmArgumentProviders rather than jvmArgs because several modules assign jvmArgs wholesale, which would discard it. Measured on a 14-core/20-thread host with `:grails-core:test --rerun-tasks`, daemons stopped between runs, comparing second runs: baseline first: baseline 3m25s, treatment 2m45s (19.5% faster) treatment first: treatment 2m24s, baseline 3m05s (22.2% faster) Running the treatment first rules out filesystem-cache ordering bias. Peak JVM count fell from 29 to 24; projected per-fork JVM threads fell from 31 to 5, so projected total threads fell from roughly 465 to 50. `-PmaxTestParallel` still overrides the default, and now does so in grails-forge as well, where it was previously ignored. Assisted-by: claude-code:claude-opus-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 8.0.x #16158 +/- ##
==================================================
+ Coverage 52.3252% 52.3323% +0.0071%
- Complexity 18536 18547 +11
==================================================
Files 2039 2039
Lines 97498 97521 +23
Branches 17138 17143 +5
==================================================
+ Hits 51016 51035 +19
- Misses 38998 39000 +2
- Partials 7484 7486 +2 🚀 New features to boost your workflow:
|
CI measurement (first draft run)All 59 jobs passed. Comparing the
Whole-workflow wall clock was 95.5 min against a ~2h14m median. How much to trust thisNot much yet, on its own. This is one run against a three-run baseline, and per-job variance on GitHub runners is large: baseline What makes the direction plausible rather than noise is that the local A/B benchmark was run in both orderings on an idle machine, and the improvement held each way (19.5% with baseline first, 22.2% with this branch first). What is actually being measured hereCI fork counts are unchanged by this PR - the The macOS result is the most interesting one for day-to-day work, since most committers develop on macOS and GitHub's macOS runners are the smallest at 3 vCPU / 7 GB. Suggested next stepRe-run this workflow a couple more times before drawing conclusions, so each job has a comparable sample count to the baseline. |
CI measurement, second sample - correcting the firstA second run of the same commit landed (59/59 jobs green again). It does not reproduce the improvement reported above, and the earlier numbers should be treated as retracted.
What this actually showsRunner variance dominates. macOS JDK 21 moved from 62.5 to 89.9 minutes on identical code - a 35 percentage point swing between two runs. Any single-sample CI comparison on this workflow, including my first one, is noise. The honest reading of two samples is that CI wall clock is roughly unchanged. Windows JDK 25 shard 0 is the one consistent signal, slower in both samples (+23.7%, +25.3%). Two samples is still thin, but it is the only job where both point the same way, so it deserves attention rather than dismissal. Worth noting the baseline for that job spans 66.8-93.7 minutes across three runs, so even this may be variance. Does this invalidate the change?Not the local result, which is the stronger evidence and was measured under controlled conditions - idle machine, daemons stopped between runs, and critically run in both orderings so filesystem-cache bias pointed against the change in one of them: That reproducibility is what a shared GitHub runner cannot offer. It is also worth restating what CI is even exercising here: fork counts are unchanged on CI by design ( The developer-machine problem this PR exists to fix - 23 JVMs, 12.4 GB, and roughly 465 threads on a 20-thread box - is unaffected by any of this. Suggested next stepIf CI timing is a merge criterion, this needs several more samples per job to say anything, particularly for Windows shard 0. If it is not, the local A/B plus unchanged CI fork counts should be sufficient. |
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Reduces workstation/CI contention during Gradle test execution by lowering local test parallelism and capping each forked JVM’s perceived processor count, plus fixes an ordering bug that prevented a specific “isolated” test task from being serialized.
Changes:
- Reduce local
maxParallelForksdefaults (while keeping CI fork budgets unchanged) and honor-PmaxTestParallelconsistently. - Add a
CommandLineArgumentProviderthat injects-XX:ActiveProcessorCount=...to cap per-fork JVM ergonomics (GC/JIT thread pools). - Fix
isolatedTestsTwoconfiguration ordering so itsmaxParallelForks = 1override is not overwritten.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
build.gradle |
Lowers local fork default and adds ActiveProcessorCountArgumentProvider to all Test tasks. |
grails-gradle/build.gradle |
Mirrors root behavior in the separate grails-gradle build; adds the same JVM processor cap. |
grails-forge/build.gradle |
Adds the JVM processor cap for forge tests and minor DSL cleanup. |
grails-forge/gradle/test-config.gradle |
Ensures -PmaxTestParallel is honored for maxParallelForks. |
grails-test-suite-uber/build.gradle |
Moves isolatedTestsTwo overrides after configureEach to prevent being overwritten. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| } | ||
|
|
||
| /** Caps each test fork's view of the machine. See the root build.gradle for the rationale. */ | ||
| final class ActiveProcessorCountArgumentProvider implements CommandLineArgumentProvider { |
| } | ||
|
|
||
| /** Caps each test fork's view of the machine. See the root build.gradle for the rationale. */ | ||
| final class ActiveProcessorCountArgumentProvider implements CommandLineArgumentProvider { |
| // Half the LOGICAL processors, not 3/4. This separate Gradle build mirrors the root | ||
| // build.gradle - see it for the full rationale. CI keeps its existing budget. | ||
| configuredTestParallel = findProperty('maxTestParallel') as Integer ?: | ||
| (isCiBuild ? 3 : Math.max(1, (Runtime.runtime.availableProcessors() / 2) as int)) |
Forked test JVMs in all three builds now fail if the processor cap is missing. grails-forge uses the same CI fork cap as grails-gradle. Assisted-by: Cursor Grok 4.6
|
Follow-up landed in
Still out of scope: memory budget, 5 GB daemon on 7 GB macOS runners, Testcontainers-per-fork, and unbenchmarked |
Avoid Gradle 10's removal of implicit parent-project lookup when test scripts read the fork-count property defined on each build root. Assisted-by: Cursor Grok 4.6
|
Follow-up |
✅ All tests passed ✅🏷️ Commit: ccca92f Learn more about TestLens at testlens.app. |
|
CI on Closing the shard 0 question from the description. This run:
Shard 0 stays roughly 2x shards 1 and 2, which is the workload difference, not a regression: shard 0 runs The new forked-JVM assertions passed everywhere they run, so the flag is landing on real CI runners and not just locally. |
The problem
A plain
./gradlew buildcan leave a developer's workstation unusable. Observed on a 14-core / 20-thread machine: 23 JVMs, 12.4 GB, with the desktop unresponsive.Three multipliers stack, and each looks reasonable on its own.
1.
availableProcessors()counts logical processors. SMT threads on x64, and on Apple silicon every efficiency core as well as every performance core.* 3 / 4of that is already measured against an inflated number - 15 forks on a 14-core host, and on an M-series Mac most of those forks land on efficiency cores.2.
maxParallelForksis per Test task. Withorg.gradle.parallel=true, several test-bearing modules run concurrently, so the real ceiling is Gradle's worker-lease pool (--max-workers, default =availableProcessors), not any single task's fork count.3. Each forked JVM sizes its own thread pools for the whole machine, because no fork knows the others exist. Measured on JDK 21 at 20 visible processors:
ParallelGCThreadsConcGCThreadsCICompilerCountThat is 31 threads per fork before a single test runs. Gradle cannot see it: it charges each fork one worker lease, as though a fork were a single thread. Gradle's own default for
Test.maxParallelForksis 1 for precisely this reason - raising it opts out of the protection Gradle already provides.15 forks x 31 threads is roughly 465 threads contending for 20 hardware threads, plus 15 x 1 GB heaps (2 GB in
grails-test-suite-persistence) against a 5 GB daemon. That is the thrashing.The oversubscription ratio scales with the machine
This is why the problem is felt on workstations and barely registers on CI. Each concurrent fork sizes its pools for every visible processor, so the "virtual processors" a build demands is forks x visible processors, against however many the host actually has:
The change normalises every host to roughly 2x. The benefit is therefore proportional to how oversubscribed the host was to begin with: dramatic on a large developer machine, modest on a small CI runner.
The change
Applied to all three builds in this repository - root,
grails-gradleandgrails-forgeeach have their ownsettings.gradle, so the duplication is unavoidable.isCiBuild ? 4in root,? 3ingrails-gradleand now also ingrails-forge).availableProcessors / maxWorkerCount, floored at 2. The denominator is the build-wide worker limit rather than any one task'smaxParallelForks, because that is what actually bounds how many forks run at once. A floor of 2 keeps G1 rather than dropping to Serial GC. That also means-PmaxTestParallel=1still advertises 2 processors to the single fork whenmaxWorkerCountis the usual unrestricted local pool; it is not a "give the one fork the whole machine" switch.Supplied via
jvmArgumentProvidersrather thanjvmArgs, because several modules assignjvmArgswholesale and would discard it.Also fixes
grails-forge, where-PmaxTestParallelwas silently ignored.Measured result
Full build
This is the workload the PR exists for.
DO_NOT_CACHE_TESTS=1 ./gradlew build --continue --profile --console=plain, on a 14-core / 20-thread i7-12800H with 63.7 GB, Gradle daemons stopped before each run, and the treatment run first so that filesystem-cache warmth works against the change:b964e60)13c9641)Thread and memory figures here are live samples taken every 2 seconds for the duration of each build, not projections.
Peak JVM count barely moves, and that is the expected result. With
org.gradle.parallel=trueand noorg.gradle.workers.max, the number of concurrent forks is bounded by the worker-lease pool, not by any single task'smaxParallelForks; lowering 15 → 10 mostly just frees leases for other modules to take. What changes is the thread count inside each fork. 1,967 versus 2,739 live threads against 20 hardware threads is essentially the whole effect.The
--profilereports confirm where the time goes: 83% of the reduction in task time is intest/integrationTesttasks.:grails-data-hibernate7-core:test:grails-fields:test:grails-gsp:test:grails-datamapping-core-test:test:grails-datamapping-support:test:grails-datastore-core:test:grails-data-mongodb-core:testAggregate
test+integrationTesttask time fell from 21h 07m to 13h 57m. These are per-task wall-clock times in a parallel build, so they overlap and sum to far more than the elapsed time; treat them as a relative indicator.Caveats, stated plainly:
groovydoc/javadocwork, so of the 25-minute gap perhaps 4-5 minutes is task-set and cache-state noise rather than this change.exit 1on flaky integration tests,:grails-test-examples-mail:integrationTestin both. With--continueneither build was shortened by them.Single module
:grails-core:test --rerun-tasks, daemons stopped between runs, comparing second runs:Running this branch first rules out filesystem-cache ordering bias - the advantage held in both directions.
A single module understates the effect. With only one
Testtask in flight there is no cross-module contention to remove, so this measures little more than the fork-count reduction:ActiveProcessorCountPer-fork thread figures are a computed projection from
java -XX:+PrintFlagsFinal -version, not a live thread count.Second commit:
isolatedTestsTwowas not actually isolatedgrails-test-suite-uber/build.gradlesetmaxParallelForks = 1onisolatedTestsTwo, but registered that block before thetasks.withType(Test).configureEachblock in the same script. Gradle runs both as deferred configuration actions in registration order, so the laterconfigureEachoverwrote it and the task ran fully parallel. Its test patterns have been deliberately serialized since 2013 because they are order sensitive.Moving the override after the
configureEachblock fixes it. CI impact is negligible: the task filters three classes and sharding assigns it to a single shard.Third commit: prove the flag lands, and cap forge CI forks
0f5f5b653aadds a forked-JVM assertion in each of the three builds. The test readsManagementFactory.getRuntimeMXBean().getInputArguments()and fails if-XX:ActiveProcessorCountis missing, and checks thatRuntime.availableProcessors()equals the advertised count. That is the regression that would catch a future wholesalejvmArgs = ...assignment swallowing the provider.Those tests passed locally:
:grails-core:test --tests grails.util.ActiveProcessorCountForkTestsgrails-gradle::grails-gradle-plugins:test --tests org.grails.gradle.plugin.core.ActiveProcessorCountForkSpecgrails-forge::grails-forge-core:test --tests org.grails.forge.ActiveProcessorCountForkSpecgrails-forgenow uses the sameconfiguredTestParallelformula asgrails-gradle(isCiBuild ? 3, otherwise half the logical processors, still honouring-PmaxTestParallel). Previously forge always usedprocessors.intdiv(2)with no CI cap.Measuring this on CI
Baseline for
gradle.ymlon8.0.x: median 2h 14m, p90 3h 52m, with core build jobs at 1h 20m - 1h 40m. Draft PRs do trigger the workflow, so this PR's own run is the measurement.CI fork counts are unchanged on root and
grails-gradle, so the only variable on those jobs is the per-fork processor cap: on a 4-vCPU runner each of the 4 forks goes from seeing 4 processors to seeing 2. Per the ratio table above that is 4x oversubscription down to 2x, against 20x down to 2x locally. CI is therefore expected to show a much smaller effect than the local full-build numbers, and one that is easily buried by runner variance - which is what the two samples collected so far look like.Windows JDK 25 shard 0 was slower in both samples (+23.7%, +25.3%). That job is not comparable to shards 1 and 2: it runs
build :grails-shell-cli:installDist groovydocplus shard 0 tests, while shards 1 and 2 run onlytestShard.groovydocandinstallDistrun in the Gradle daemon, which does not receive-XX:ActiveProcessorCount, so those extra tasks cannot be attributed to the processor cap. The remaining plausible causes are Windows cache-writer variance plus the shard 0 tests seeing 2 processors instead of 4. That is not a reason to revert the cap: it is the expected cost of the 4x → 2x change on the least contention-bound job in the matrix. Worth watching on the next CI run of this commit, not a blocker.Follow-ups, deliberately not in this PR
org.gradle.workers.maxor a sharedBuildService.withReuse, so N forks means N containers - and on macOS and Windows that memory comes from a Docker Desktop VM the JVM cannot see.grails-gradleandgrails-forgehave not been benchmarked. They are separate builds; the full-build numbers above cover the root build only.