ci: write releases from one paced job, not from 107 at once by widgetii · Pull Request #121 · OpenIPC/builder

widgetii · 2026-08-17T04:14:28Z

Every matrix job uploads its images to three shared releases with softprops/action-gh-release, so a nightly fires on the order of 400 concurrent asset writes at nightly, latest and the dated tag.

OpenIPC/firmware ran exactly this design and the releases API answered HTTP 500 under the contention — run 27108181857 built every board and still went red on 14 jobs, all of them in their upload step. Firmware then hit the sequel: consolidating into one job but keeping concurrent uploads tripped the per-actor secondary rate limit at 350 of 393 assets. This lands where that ended up, without the two failures in between.

What changes

Matrix jobs stage what they produced into dist/ and hand it to actions/upload-artifact. A single publish job downloads all of it and drives the release writes with gh:

one asset at a time, paced ~1/s under the per-minute content-mutation ceiling
exponential backoff (15s doubling, 6 attempts) so a transient 403/429 is retried rather than failing the nightly
--clobber on upload, tag refs moved with gh api -X PATCH

The single-writer property is what fixes the 500s; the pacing is what keeps that fix from tripping the other limit. Best-effort by design: it publishes whatever the matrix produced even if some devices failed, which is what enrich_manifest.py already assumes.

Asset routing

enrich_manifest.py only ever reads dated releases (list_dated_releases() → TAG_RE = ^nightly-\d{8}-[0-9a-f]{7}$), and only parses .tgz names.

release	gets
`nightly-YYYYMMDD-<sha>`	everything — images + `sizes.*.json`
`nightly`, `latest`	images only

Re-uploading the size sidecars to two more tags is pure rate-limit cost: nothing reads them from those tags.

Asset names are unchanged

Staging copies whichever of the two naming schemes the build used, so names are byte-identical to today:

device	scheme	filename
`gk7205v200_fpv` (1 underscore)	simple	`openipc.gk7205v200-nor-fpv.tgz`
`ssc338q_apfpv`	simple	`openipc.ssc338q-nor-apfpv.tgz`
`t31_lite_wyze-v3b` (2+)	compound	`t31_lite_wyze-v3b-nor.tgz`
`hi3518ev200_lite_switcam-hs303-v2`	compound	`…-nand.tgz`

Verified every one of them still parses through enrich_manifest.py's COMPOUND_RE / SIMPLE_RE — all six round-trip to the right (platform, flash) pair.

Also

ci-gate now covers publish, so a failed release write fails the run instead of passing quietly.
The staging loop uses if rather than a trailing && chain. The step runs under bash -e, where a chain that tests false is only harmless because something after it happens to reset the status — I'd rather that not be load-bearing.

Scope note

I'd said this PR would also record the firmware SHA in the nightly notes, as groundwork for narrowing the nightly by upstream diff. I dropped it. Recording a SHA resolved at the start of the run would be a half-truth: each of the 107 jobs clones OpenIPC/firmware at HEAD independently, so a nightly can already contain devices built from different firmware commits. Making it truthful means pinning every job via OPENIPC_FW_REV, which costs a full clone instead of --depth=1 across 107 jobs. That cost is only worth paying if the nightly-narrowing actually pays off, so it belongs in that change, measured, rather than smuggled in here.

Testing

This is a master.yml-only change, so CI narrows it to the 15-device smoke set. Note that the publish path itself cannot run on a PR (it's guarded on github.event_name != 'pull_request', and there are no artifacts) — so the real exercise is the first nightly after merge, or a workflow_dispatch. What is verified here offline: the DATED/IMAGES split, the staging step against both naming schemes and against missing NAND/sizes, the notes block, and the manifest round-trip.

🤖 Generated with Claude Code

Every matrix job uploaded its images to three shared releases with softprops/action-gh-release, so a nightly fired on the order of 400 concurrent asset writes at `nightly`, `latest` and the dated tag. OpenIPC/firmware ran exactly this design and the releases API answered HTTP 500 under the contention -- run 27108181857 built every board and still went red on 14 jobs, all of them in their upload step. Firmware then hit the sequel: consolidating into one job but keeping concurrent uploads tripped the per-actor secondary rate limit at 350 of 393 assets. This lands where that ended up, without the two failures in between. Matrix jobs now stage what they produced into dist/ and hand it to actions/upload-artifact. A single publish job downloads all of it and drives the release writes with gh: one asset at a time, paced ~1/s under the per-minute mutation ceiling, with exponential backoff so a transient 403/429 is retried rather than failing the nightly. The single-writer property is what fixes the 500s; the pacing is what keeps the fix from tripping the other limit. The dated release keeps the full asset set, because enrich_manifest.py only ever reads dated releases. `nightly` and `latest` are delivery aliases for flashers and now get images only: re-uploading the size sidecars to two more tags is pure rate-limit cost, and nothing reads them from those tags. Staging copies whichever of the two naming schemes the build used -- compound devices rename to <device>-<flash>.tgz in the workspace root, single-underscore ones leave openipc.<soc>-<flash>-<variant>.tgz under openipc/output/images/ -- so asset names are byte-identical to today. Checked both against enrich_manifest.py's COMPOUND_RE and SIMPLE_RE. ci-gate covers publish, so a failed release write fails the run instead of passing quietly. The staging loop uses `if` rather than a trailing `&&` chain: the step runs under `bash -e`, where a chain that tests false is only harmless because something after it happens to reset the status. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

qodo-free-for-open-source-projects · 2026-08-17T04:15:12Z

PR Summary by Qodo

CI: publish GitHub releases from a single paced job

⚙️ Configuration changes ✨ Enhancement 🕐 20-40 Minutes

AI Description

• Stage per-device firmware outputs into artifacts instead of writing releases from the matrix
• Add a single publish job to upload assets sequentially with pacing and exponential backoff
• Extend ci-gate to include publish so release failures fail the workflow

Diagram

graph TD
  A["Matrix build jobs"] --> B["Stage to dist/"] --> C["upload-artifact (fw-*)"] --> D[("Workflow artifacts")]
  D --> E["publish job"] --> F["gh release create/edit"] --> G["GitHub Releases (tags/assets)"]
  E --> H["gh api PATCH tag refs"] --> G
  E --> I["paced upload + retries"] --> G
  E --> J["ci-gate"]

High-Level Assessment

The following are alternative approaches to this PR:

1. Keep action-gh-release but serialize via a separate upload workflow

➕ Less custom bash/gh scripting inside the workflow
➕ Could reuse existing action behavior for release creation/notes
➖ Still hard to guarantee single-writer behavior without careful coordination
➖ Doesn’t address per-asset concurrency/throttling as directly as explicit gh pacing

2. Bundle outputs into fewer assets (per-device tar/zip, or a single archive)

➕ Dramatically reduces number of release asset mutations and rate-limit exposure
➕ Simplifies upload loop and speeds publishing
➖ Changes artifact distribution format for consumers (may break tooling expecting per-image assets)
➖ Reduces convenience for users downloading a single device image

3. Publish only to artifacts (skip GitHub Releases for nightlies)

➕ Avoids Releases API contention and secondary rate limits entirely
➕ Artifacts are designed for high-volume CI output
➖ Worse UX for ‘nightly/latest’ consumption; artifacts are less discoverable and expire
➖ Would require changes to any downstream tooling/users expecting Releases tags

Recommendation: The PR’s approach (single publish job + sequential paced gh uploads + retry/backoff) is the best fit given the requirement to keep GitHub Releases as the distribution channel and preserve existing asset names. The explicit single-writer property addresses the observed HTTP 500 contention, and the 1/s pacing + exponential backoff is a pragmatic mitigation for secondary rate limits. If rate limits remain an issue long-term, the next step to consider is reducing asset count via bundling, but that has compatibility tradeoffs.

Files changed (1) +150 / -39

Other (1) +150 / -39

master.ymlCentralize release publishing into a paced single-writer job +150/-39

Centralize release publishing into a paced single-writer job

• Replaces per-matrix-job GitHub Release uploads with a two-phase flow: matrix jobs stage outputs into dist/ and upload them as artifacts. Adds a dedicated publish job that downloads all artifacts and writes to GitHub Releases sequentially using gh with pacing, clobbering, tag-ref updates, and exponential backoff retries. Updates ci-gate to depend on publish and fail the run on real publish failures while allowing expected skips.

.github/workflows/master.yml

qodo-free-for-open-source-projects · 2026-08-17T04:17:48Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. ~~Zero-count check broken~~ ✓ Resolved 🐞 Bug ≡ Correctness

Description

Collect assets uses wc -l, which emits leading whitespace; steps.collect.outputs.count can
become '      0' and still pass the != '0' check, causing the publish script to run even when no
artifacts were downloaded. In that case it can still create/edit releases and force-move the
nightly/latest tag refs while uploading nothing.

Code

.github/workflows/master.yml[R349-352]

+          mkdir -p dist
+          count=$(find dist -type f | wc -l)
+          echo "Collected ${count} asset(s):"
+          ls -la dist || true

Evidence

The workflow writes the raw wc -l output into GITHUB_OUTPUT and then performs a literal string
comparison against '0'. If the directory is empty, the padded wc output can be non-equal to
'0', and the subsequent script still edits releases and force-updates tag refs even if no uploads
occur.

.github/workflows/master.yml[349-353]
.github/workflows/master.yml[361-363]
.github/workflows/master.yml[418-431]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The publish step is gated by `steps.collect.outputs.count != '0'`, but `count` is computed via `wc -l`, which outputs padded numbers (leading spaces). This can make an empty asset set look non-zero (`"      0"`), triggering the publish step and moving tags without uploading assets.
### Issue Context
This occurs in the `publish` job:
- `count=$(find dist -type f | wc -l)`
- `echo "count=${count}" >> "$GITHUB_OUTPUT"`
- `if: steps.collect.outputs.count != '0'`
### Fix Focus Areas
- .github/workflows/master.yml[349-363]
### Suggested fix
- Normalize `count` to a clean integer before writing to `$GITHUB_OUTPUT`, e.g.:
- `count=$(find dist -type f | wc -l | tr -d '[:space:]')`
- or `count=$(find dist -type f -print | awk 'END{print NR+0}')`
- Optionally add a defensive check inside the publish script (before any `ensure_release` / tag updates): if no files are found, print a notice and exit 0.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~Artifact download errors hidden~~ ✓ Resolved 🐞 Bug ☼ Reliability

Description

actions/download-artifact is configured with continue-on-error: true, so transient/real download
failures can produce an empty or partial dist/ without failing the publish job. Because
ci-gate treats publish=success|skipped as OK, the run can go green while releases remain stale
or incomplete.

Code

.github/workflows/master.yml[R338-341]

+      - name: Download build artifacts
+        uses: actions/download-artifact@v4
+        continue-on-error: true
+        with:

Evidence
The download step explicitly continues on error, and the gate logic only fails when publish is
neither success nor skipped—so a publish job that did nothing due to download failure can still pass
branch protection.
.github/workflows/master.yml[338-345]
.github/workflows/master.yml[346-354]
.github/workflows/master.yml[475-479]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The publish job suppresses failures from `actions/download-artifact@v4` via `continue-on-error: true`. This can allow CI to pass without actually downloading (and therefore publishing) the artifacts.
### Issue Context
- The download step is allowed to fail without failing the job.
- `ci-gate` accepts `needs.publish.result` as `success` or `skipped`, so a silently-successful publish job masks release publication problems.
### Fix Focus Areas
- .github/workflows/master.yml[338-345]
- .github/workflows/master.yml[475-479]
### Suggested fix
- Remove `continue-on-error: true` so artifact download failures fail the job.
- If you need to tolerate the specific case of "no artifacts exist" while still failing on real download errors:
- Prefer using a first-class option on `download-artifact` (if available in your pinned version) to ignore missing artifacts, rather than swallowing all errors.
- Or add an explicit follow-up step that checks `steps.<download_id>.outcome`/`conclusion` and fails the job when the download action errored, while keeping the "no files" case as a clean skip (based on the normalized count from the collect step).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Tip of the day

💡 Did you know, you can route each action level your way: inline, summary, both, or drop

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Two High findings on #121, both real. The download step carried continue-on-error: true, so a transient or real artifact-download failure produced an empty dist/, the publish step's count guard skipped it, the job went green, and ci-gate accepts publish=success. A nightly could silently write nothing while the run stayed green. The flag stays -- it exists so a matrix where every device failed, and so uploaded no fw-* artifact at all, does not hard-fail before the gate can report why -- but Collect now turns it back into a failure for every other case: if the matrix went green, a download that did not succeed is an error, and zero collected assets is an error. Only "matrix failed and produced nothing" is still tolerated, and the gate fails that run on the matrix result anyway. The second finding said count could read " 0" because wc pads. On GNU coreutils, which is what the runner has, it does not -- that is BSD behaviour, and `printf '' | wc -l` here gives an unpadded 0. Normalised anyway, since it costs a tr and stops the guard depending on which wc is in front of it. Chasing it did surface a worse hole it sits next to. dist/ can hold sidecars and no images -- one device that produced a size report and no .tgz is enough -- and count is then 1, so publish ran, force-moved the nightly and latest tag refs, and uploaded zero images. The two tags users flash from would point at a build with no firmware behind them. Both tags now move only when there is at least one image to move them to; otherwise they stay on yesterday's, with a warning. Checked all four states offline, since none of this path can run on a pull request: green matrix with a broken download, green matrix with an empty dist, failed matrix with nothing produced, and a normal night. The first two now fail and used to pass. Note the same continue-on-error and unnormalised wc are in OpenIPC/firmware's publish job, which is where this one came from. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Review on OpenIPC/builder#121 found two holes in its publish job. That job was seeded from this one, and both holes are still here. The artifact download runs with continue-on-error, so a transient or real download failure produced an empty dist/, the count guard skipped the publish step, the job went green, and ci-gate accepts publish=success -- a nightly could silently write nothing while the run stayed green. The flag stays, because it is load-bearing for exactly one case: a matrix where every board failed uploads no fw-* artifact at all, and a hard download failure there would mask the real reason in the gate. Collect now turns it back into a failure for every other case: a green matrix with a download that did not succeed is an error, and a green matrix with zero collected assets is an error. Only "matrix failed and produced nothing" is still tolerated, and the gate fails that run on the matrix result anyway. The nightly and latest tag moves were unconditional once any asset existed. dist/ can hold sidecars and no images -- one board producing a size report and no .tgz is enough -- and the script would force-move the two tags flashers pull from to a build with no firmware behind them. Both now move only when there is at least one image to move them to; otherwise they stay on yesterday's build, with a warning. The count is also normalised through tr. On the runner's GNU coreutils wc does not pad, so this was not exploitable here -- padding is BSD behaviour -- but the guard should not depend on which wc is in front of it. While in the area, ci-matrix.py now writes $GITHUB_OUTPUT itself instead of documenting that it does while relying on the workflow to redirect stdout (the docstring inaccuracy was also flagged on builder). Under a redirect every print() is one keystroke away from corrupting the step outputs, and a crash between the first line and the last leaves a half-written file that Actions still reads. Falls back to stdout when the variable is unset; --stdin never touches the file. All four collect states and all output paths checked offline; none of the publish path can run on a pull request. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

widgetii requested review from cronyx, flyrouter and viktorxda as code owners August 17, 2026 04:14

qodo-free-for-open-source-projects Bot reviewed Aug 17, 2026

View reviewed changes

Comment thread .github/workflows/master.yml

Comment thread .github/workflows/master.yml

widgetii merged commit 5701c67 into master Aug 17, 2026
19 checks passed

widgetii deleted the ci-single-publish branch August 17, 2026 04:57

widgetii mentioned this pull request Aug 17, 2026

ci: carry the publish hardening back from builder OpenIPC/firmware#2279

Merged

Navigation Menu

Uh oh!

Uh oh!

ci: write releases from one paced job, not from 107 at once - #121

ci: write releases from one paced job, not from 107 at once#121
widgetii merged 2 commits into
masterfrom
ci-single-publish

widgetii commented Aug 17, 2026

Uh oh!

qodo-free-for-open-source-projects Bot commented Aug 17, 2026

Uh oh!

qodo-free-for-open-source-projects Bot commented Aug 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

widgetii commented Aug 17, 2026

What changes

Asset routing

Asset names are unchanged

Also

Scope note

Testing

Uh oh!

qodo-free-for-open-source-projects Bot commented Aug 17, 2026

PR Summary by Qodo

Uh oh!

qodo-free-for-open-source-projects Bot commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

qodo-free-for-open-source-projects Bot commented Aug 17, 2026 •

edited

Loading