Skip to content

Shard generated tests across GitHub runners - #2070

Open
nogates wants to merge 1 commit into
masterfrom
nogates/split-test-matrix
Open

Shard generated tests across GitHub runners#2070
nogates wants to merge 1 commit into
masterfrom
nogates/split-test-matrix

Conversation

@nogates

@nogates nogates commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • reuse the single compiled Cucumber binary across four GitHub Actions matrix jobs
  • assign every scenario deterministically to exactly one shard
  • preserve unsharded behavior when the shard environment variables are absent
  • revert the temporary src/lib.rs benchmark comments

Baseline

The latest unsharded run executed 1,682 scenarios. Its Run tests step took 5m55s and the complete test job took 6m12s.

Validation

  • cargo fmt -- --check
  • CARGO_INCREMENTAL=0 cargo check --test main
  • CARGO_INCREMENTAL=0 cargo test --test main --no-run
  • standalone sharding unit tests (2 passed)
  • workflow YAML parse and git diff --check

@nogates

nogates commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

First CI measurement is green and covers all 1,682 scenarios exactly once:

Shard Scenarios Run tests Complete job
1 394 1m30s 1m45s
2 438 1m31s 1m48s
3 441 1m30s 1m43s
4 409 1m04s 1m17s

The previous unsharded test step/job took 5m55s/6m12s. The slowest shard is now 1m31s/1m48s, reducing the test phase critical path by 4m24s at the step level and 4m24s at the job level.

Despite a slower shared compile in this particular run (4m56s versus 3m49s), the complete workflow improved from 11m10s to 7m34s, a 3m36s wall-clock reduction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant