Skip to content

ci: balance runner pools across vendors and persist fuzz corpus - #1838

Open
simongdavies wants to merge 2 commits into
mainfrom
simongdavies-ci-pool-balance-fuzz-fixes
Open

simongdavies wants to merge 2 commits into
mainfrom
simongdavies-ci-pool-balance-fuzz-fixes

Conversation

@simongdavies

@simongdavies simongdavies commented Sep 17, 2026

Copy link
Copy Markdown
Member

Runner pools

Vendor-agnostic jobs select AMD or Intel pools by run-ID parity, following copilot-setup-steps.yml. Other runner labels and JobId values are preserved. Coverage keeps one job. Explicit vendor matrices, benchmark pools and arm64 runner labels are unchanged.

Fuzzing

  • Corpus restore, minimisation and save are opt-in through persist_corpus. Only the weekly workflow enables them, including its manual runs.
  • Per-target caches reuse earlier inputs. cargo fuzz cmin reduces redundant inputs with a 20-minute limit. Trace targets use matching trace guests. Saving after failure or cancellation is best-effort, and minimisation does not enforce a storage cap.
  • Callers supply timeout_minutes: 330 for weekly runs, 25 for PRs and 35 for daily arm64 runs.
  • Failure notifications include timed-out dependencies. !cancelled() suppresses notifications for deliberate workflow cancellation.

Vendor-agnostic jobs pick their 1ES pool by run-ID parity, so work spreads
across the AMD and Intel pools instead of queueing on AMD alone. Pools where
the vendor is semantically required keep an explicit vendor.

Fuzz runs cache the corpus per target, so coverage-guided fuzzing builds on
the previous week rather than starting from empty on a single-use runner.
A cmin pass bounds corpus growth against the repo cache limit. The fuzz job
carries an explicit timeout derived from the libFuzzer budget, and a
cancelled run notifies as well as a failed one.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fuzz timeout, cancellation notification, architecture-specific caching, and cache retention issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR balances CI runner pools across AMD and Intel vendors and adds persistent, minimized fuzzing corpora with timeout and cancellation handling.

Changes:

  • Selects runner pools by run-ID parity.
  • Adds fuzz corpus caching and minimization.
  • Adds explicit fuzz timeouts and cancellation notifications.
File summaries
File Description
Justfile Adds corpus minimization recipes.
.github/workflows/PrimeCaches.yml Balances cache jobs across vendors.
.github/workflows/Fuzzing.yml Handles cancelled fuzz jobs.
.github/workflows/dep_update_guest_locks.yml Balances lock-update runners.
.github/workflows/dep_fuzzing.yml Adds timeout, corpus caching, and minimization.
.github/workflows/dep_code_checks.yml Balances Linux and Windows checks.
.github/workflows/dep_build_guests.yml Balances guest-build runners.
.github/workflows/CreateRelease.yml Balances release runners.
.github/workflows/Coverage.yml Selects coverage CPU vendor by run ID.
.github/workflows/CargoPublish.yml Balances publishing runners.
Review details

Suppressed comments (3)

.github/workflows/dep_fuzzing.yml:31

  • This reusable workflow is also called by DailyArm64.yml, so this timeout applies to the arm64 fuzz jobs. That workflow's notifier still checks only needs.fuzzing.result == 'failure'; a timeout reports cancelled, so an arm64 timeout will silently skip the release-blocking notification. Update that caller's condition to handle dependency cancellation while excluding a human-cancelled run.
    timeout-minutes: ${{ inputs.max_total_time / 60 + 30 }}

.github/workflows/dep_fuzzing.yml:73

  • The corpus cache is keyed only by target, but this reusable workflow runs the same targets on both X64 and arm64. The prefix restore can therefore feed an architecture-specific minimized corpus to the other architecture, and each architecture's cmin pass can discard inputs that are only useful on the other one. Include inputs.arch in both the save key and restore prefix.
          key: fuzz-corpus-${{ inputs.target }}-${{ github.run_id }}-${{ github.run_attempt }}
          restore-keys: |
            fuzz-corpus-${{ inputs.target }}-

.github/workflows/dep_fuzzing.yml:73

  • Including github.run_id makes every run create a new immutable cache entry. cmin bounds each snapshot but does not remove prior weekly or pull-request snapshots, so the six targets can still consume the 10 GB cache budget over time and evict the Rust caches this PR is intended to protect. Use a bounded retention or deletion strategy, or storage that updates one corpus per target.
          key: fuzz-corpus-${{ inputs.target }}-${{ github.run_id }}-${{ github.run_attempt }}
          restore-keys: |
            fuzz-corpus-${{ inputs.target }}-
  • Files reviewed: 10/10 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/workflows/dep_fuzzing.yml Outdated
Comment thread .github/workflows/Fuzzing.yml Outdated
Actions expressions have no arithmetic operators, so the fuzz job timeout
comes from callers as a precomputed timeout_minutes input. The invalid
expression stopped the whole workflow from parsing, which surfaced as
Fuzzing and Validate Pull Request creating no jobs at all.

Corpus caching and minimisation run only where they pay off, under the
persist_corpus input. The weekly scheduled run enables it. PR and daily
arm64 runs fuzz for 5 minutes, so persisting a corpus there would spend the
shared 10 GB cache budget and evict the rust-cache entries PrimeCaches.yml
maintains.

notify-failure guards on !cancelled() so a cancelled run files no issue. A
job exceeding its own timeout still reports 'cancelled' while the run is
not cancelled, so timeouts continue to notify.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
@simongdavies simongdavies added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/enhancement For PRs adding features, improving functionality, docs, tests, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants