Skip to content

ci: consolidate release onto GitHub Actions and drop Woodpecker - #185

Open
slayerjain wants to merge 3 commits into
mainfrom
ci/migrate-off-woodpecker
Open

ci: consolidate release onto GitHub Actions and drop Woodpecker#185
slayerjain wants to merge 3 commits into
mainfrom
ci/migrate-off-woodpecker

Conversation

@slayerjain

@slayerjain slayerjain commented Jul 31, 2026

Copy link
Copy Markdown
Member

Why

This repo was running two CI systems in parallel. Some of it was pure duplication, and the rest was split across the two in a way that made releases fragile.

The build pipeline was 100% redundant. .woodpecker/build.yml and .github/workflows/java-agent.yml ran identical commands on an identical JDK 8/17/21 matrix (mvn -B -DskipTests clean verify + ./scripts/smoke-javaagent.sh). Every PR and every push to main built the project twice. This was introduced by accident: f70e488 deliberately migrated off GitHub Actions to Woodpecker, then 8d48e62 re-added GitHub Actions three days later without removing the Woodpecker half.

The release was split across both systems. Woodpecker published to Maven Central; GitHub Actions published the GitHub Release. Both fired independently off the same tag with nothing coupling them, so the jar attached to the GitHub Release was a separate, unsigned build of the one on Central. Verified on v2.1.0: the two jars have different SHA-256 sums. (The contents are fine — all 2274 entries match by CRC, the difference is only build timestamps — but consumers had no way to establish that, and no signature to check.)

Since this is a public repo, PR checks belong where contributors can actually see them.

Correction to an earlier version of this description, which claimed v2.0.6 reached GitHub but not Central. That was wrong, and worth stating plainly because it drove the original design. v2.0.6 is on Central and always was — jar, sources, pom and signatures all present. What failed on that release was the Woodpecker wait step, via the ${VERSION} templating bug fixed in c1b96c1; the deploy had already succeeded. The real incident was cosmetic. That correction changed the design (see below).

What changed

Woodpecker is removed entirely and its Central publishing is folded into the release workflow, now split into two jobs:

publish          build -> sign -> smoke test -> stage assets -> deploy to Central
github-release   wait for repo1 -> create the GitHub Release

Publishing to Central is irreversible and non-idempotent — the Portal rejects a re-upload of an existing version — so everything that can fail cheaply runs before it, and the deploy is the last step of its job. Everything after it lives in a separate job that can be re-run on its own.

This is a direct consequence of the correction above. A single job would have taken v2.0.6's cosmetic failure (deploy fine, wait step failed) and made it unrecoverable: the job would go red after an irreversible publish, and re-running it could only re-attempt a deploy Central now rejects, permanently stranding the GitHub Release. Now a slow repo1 sync fails only github-release, which re-runs independently against the staged assets carried over as a workflow artifact. workflow_dispatch is added as a manual recovery path, and a concurrency group stops two releases of the same ref overlapping (cancel-in-progress: false — cancelling mid-deploy is the one thing that could strand a release).

The jar that ships is the jar that was smoke tested. The deploy reuses target/ without cleaning, and shade is idempotent here, so the published artifact is byte-identical to the one tested — verified locally against a real signed build.

Nothing Woodpecker did is lost: GPG import, the central server credentials, mvn -P release deploy, and the repo1 poll all carried over. The ${VERSION}-vs-$VERSION workaround (c1b96c1) is gone — that was a Woodpecker templater quirk.

Hardening

  • Shell injection fixed. Git accepts backticks and semicolons in ref names and v*.*.* matches them, so an unquoted tag interpolated into the Maven arguments was injectable by anyone with push access — in a job holding the Central signing key. The version is now validated against ^[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9.]+)?$ before the key is imported.
  • Signatures are required, not optional. The previous nullglob guard meant a signature-less build would publish a green, unverifiable GitHub Release. An unmatched glob now fails the job, and pom signatures + the parent pom are staged too (the old *.jar.asc glob skipped them), so the assets are verifiable and usable offline.
  • versions-maven-plugin pinned to 2.21.0. Unqualified versions:set resolved the latest plugin from Central at run time, executing unreviewed code in the job holding the signing key.
  • -Dgpg.passphrase dropped from argvsetup-java's gpg-passphrase already wires the server entry maven-gpg-plugin reads; passing it again only exposed it in the process table.
  • Timeout budget corrected. The pom's waitMaxTime=7200 is 120 minutes on its own, so a single 150-minute cap could not cover it plus the build plus the 30-minute poll.
  • --max-time on the repo1 poll; a release left as a draft by an interrupted run is now promoted.
  • Restored CI for PRs not targeting main. Woodpecker ran the matrix on every PR regardless of target; the branches: [main] filter silently dropped coverage for this repo's long-lived integration branches.

Recovery (second review pass)

The two-job split let a failed github-release be re-run without re-deploying, but the deploy itself was not idempotent — so any whole-run retrigger (re-pushed tag, Re-run all jobs, or the workflow_dispatch advertised as the manual recovery path) re-ran mvn deploy against an already-published version, got rejected, and skipped github-release. The documented recovery path could never produce the release it existed to rescue, and Re-run all jobs additionally deletes the run's artifacts, destroying the validated assets the safe path depends on.

The deploy is now guarded by a check against Central and skipped when the version is already published, so every retrigger converges on the same end state.

  • overwrite: true on the artifact upload — v4 names are immutable within a run and partial re-runs don't clear them, so re-running a publish that failed at the deploy would have 409'd at the upload, before reaching the step that failed.
  • Repo1 poll bounded by elapsed time, not iterations. With --max-time 30 per request, 60 iterations could run a full hour against a 45-minute job timeout — the actionable "re-run this job" message would never print.
  • Version guard corrected: it rejected legal pre-release tags with a hyphen (v2.1.0-alpha-1) and accepted SNAPSHOT and leading zeros, where v02.1.0 would irreversibly publish a coordinate distinct from v2.1.0.
  • Concurrency group added to java-agent.yml so superseded PR runs cancel, now that its trigger is unfiltered.
  • Corrected the "byte for byte" claim: the agent and sources jars are identical across the two lifecycle passes (jar:jar and source:jar-no-fork skip as up-to-date); the javadoc jar differs by its embedded generation timestamp — content-equivalent, and carries its own signature.

⚠️ Before merging — repository secrets

The release workflow needs these four secrets, which currently live only in Woodpecker. Names map 1:1 (uppercased, and the space in central password fixed):

Woodpecker GitHub secret
maven_gpg_private_key MAVEN_GPG_PRIVATE_KEY
maven_gpg_passphrase MAVEN_GPG_PASSPHRASE
central_username CENTRAL_USERNAME
central password CENTRAL_PASSWORD

MAVEN_GPG_PRIVATE_KEY must be the ASCII-armored private key (gpg --armor --export-secret-keys 8541784E4EC36FB8); actions/setup-java has none of the base64 / escaped-newline fallback logic the Woodpecker script carried.

Only tag builds are affected, so merging does not break PR CI — but the next release fails if the secrets aren't in place. The failure is safe: with no key, the build dies at gpg:sign during verify, before anything is uploaded.

Housekeeping

  • Four stale secrets from June 2024 remain (GPG_PRIVATE_KEY, GPG_PASSPHRASE, MAVEN_USERNAME, MAVEN_PASSWORD) from the pre-Woodpecker OSSRH workflow. I deliberately did not reuse these names so a dead credential can't cause a confusing release failure. Worth deleting, along with the inert repository environment also named GPG_PRIVATE_KEY.
  • After merge, remove the Woodpecker repo hook and its four secrets. Those secrets are currently scoped to pull_request events on a public repo with PR builds enabled — worth confirming whether fork PRs required approval, and rotating the signing key is a cheap precaution while decommissioning either way.

Testing

All run locally on this branch:

  • mvn -B -ntp -DskipTests clean verify and ./scripts/smoke-javaagent.sh — pass (all 5 scenarios)
  • Full signed release build with a throwaway GPG key — confirms the release profile produces and signs the main, -sources, -javadoc and pom artifacts, plus the parent pom, and that the smoke test passes against the signed build
  • Confirmed setup-java's passphrase mechanism works standalone — signing succeeds with the passphrase supplied only via the gpg.passphrase server entry, no -Dgpg.passphrase
  • Confirmed the shipped jar is the tested jar — re-ran the lifecycle without clean against a signed build; identical SHA-256
  • Staging step executed verbatim against the signed build: 12 assets, every .jar/.pom with a matching .asc, all signatures verifying Good signature
  • Failure paths exercised: a missing signature and a fully absent .asc glob both exit non-zero
  • Version guard exercised: accepts v2.1.0, v2.0.6-rc1, v10.20.30; rejects v1.0.0;`id` , v1.0.0;touch$IFS/tmp/x, v1.0.0 && id, main, v1.0
  • Every run: block syntax-checked with bash -n; both workflow files parsed

The repo ran two CI systems in parallel. .woodpecker/build.yml was a
verbatim duplicate of .github/workflows/java-agent.yml — same JDK 8/17/21
matrix, same `mvn -B -DskipTests clean verify` plus smoke test — so every
PR and every push to main built the project twice.

The release side was split rather than duplicated: Woodpecker published to
Maven Central while GitHub Actions published the GitHub Release, both
triggered independently off the same tag. That split is what let v2.0.6
end up with a green GitHub Release while the Woodpecker Central publish
had failed, and it meant the jar attached to the GitHub Release was a
separate, unsigned build of the one on Central.

This is a public repo, so PR checks belong where contributors can see
them. Woodpecker is removed entirely and its Central publishing is folded
into the release workflow, which now does the whole release in one job:

  build + smoke test (pre-flight gate, unsigned)
    -> sign and deploy to Maven Central
    -> wait for the artifact on repo1
    -> create the GitHub Release from those same signed artifacts

Central publishing is irreversible and is the release that matters, so it
runs first and gates everything after it: a failed Central publish now
leaves no GitHub Release behind. The release assets are the exact files
deployed to Central, with their .asc signatures attached so they can be
verified against the release key.

The sources and javadoc jars are now copied unconditionally instead of
behind an `if -f` guard, so a release build that fails to produce them
fails the job rather than silently publishing an incomplete release.

Requires these repository secrets, previously held by Woodpecker:
MAVEN_GPG_PRIVATE_KEY, MAVEN_GPG_PASSPHRASE, CENTRAL_USERNAME,
CENTRAL_PASSWORD.

Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
Review follow-ups on the Woodpecker migration. The most important one is
that the previous version of this workflow got its own rationale wrong.

v2.0.6 is, and always was, on Maven Central — jar, sources, pom and
signatures all present. What actually failed on that release was the
Woodpecker *wait* step, via the ${VERSION} templating bug fixed in
c1b96c1. The deploy had already succeeded. So the real incident was
cosmetic, and the single-job design made that exact shape unrecoverable:
a slow repo1 sync would fail the job after an irreversible publish, and
re-running it could only re-attempt a deploy that Central would now
reject, permanently stranding the GitHub Release.

Split into two jobs so publishing and announcing fail independently:

  publish         build -> sign -> smoke test -> stage assets -> deploy
  github-release  wait for repo1 -> create the GitHub Release

Everything that can fail cheaply now runs before the point of no return,
and the deploy is the last step of its job. If Central syncs slowly, only
github-release fails and it can be re-run on its own; the staged assets
are carried between jobs as a workflow artifact so a re-run publishes the
exact bits that were validated. workflow_dispatch is added as a manual
recovery path, and a concurrency group prevents two releases of the same
ref from overlapping (cancel-in-progress is deliberately false).

The release jar is now the jar that was smoke tested. The deploy reuses
target/ without cleaning, and maven-shade-plugin is idempotent here, so
the published artifact is byte-identical to the one the smoke test ran
against — verified locally by re-running the lifecycle against a signed
build and comparing checksums.

Also:

- Validate the version against ^[0-9]+\.[0-9]+\.[0-9]+(-[A-Za-z0-9.]+)?$
  before the signing key is imported. Git accepts backticks and
  semicolons in ref names and `v*.*.*` matches them, so an unquoted tag
  interpolated into the Maven arguments was a shell injection reachable
  by anyone with push access, in a job holding the Central signing key.
- Pin versions-maven-plugin to 2.21.0. Unqualified `versions:set`
  resolved the latest plugin from Central at run time.
- Require signatures instead of tolerating their absence. The nullglob
  guard meant a signature-less build would publish a green, unverifiable
  GitHub Release; an unmatched glob now fails the job. Also stages the
  pom signatures and the parent pom, which the previous *.jar.asc glob
  skipped, so the assets are usable and verifiable offline.
- Drop -Dgpg.passphrase from argv. actions/setup-java's gpg-passphrase
  already wires the gpg.passphrase server entry that maven-gpg-plugin
  reads; passing it again only exposed it in the process table. Verified
  that signing succeeds via the settings.xml route alone.
- Add --max-time to the repo1 poll so a stalled connection cannot blow
  past the intended 30 minute bound, and promote a release left as a
  draft by an interrupted run.
- Raise the timeout budget and split it across the two jobs. The pom's
  waitMaxTime=7200 is 120 minutes on its own, so the old single 150
  minute cap could not cover it plus the build and the 30 minute poll.
- Restore CI for pull requests that do not target main. Woodpecker ran
  the matrix on every pull request regardless of target; the branch
  filter in java-agent.yml silently dropped coverage for this repo's
  long-lived integration branches.

Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
Second review pass on the release workflow. The happy path held up; the
recovery path did not.

The two-job split let a failed github-release job be re-run without
re-deploying, but nothing made the deploy itself idempotent. Any whole-run
retrigger — a re-pushed tag, "Re-run all jobs", or the workflow_dispatch
that the header comment advertises as the manual recovery path — re-ran
`mvn deploy` against a version Central had already accepted, was rejected,
and skipped github-release. The documented recovery path could therefore
never produce the release it existed to rescue. Worse, "Re-run all jobs"
deletes the run's artifacts first, so that one wrong click also destroyed
the validated, signed assets the safe path depended on.

The deploy is now guarded by a check against Central and skipped when the
version is already published, so every retrigger converges on the same
end state instead of dead-ending.

Also:

- Add overwrite: true to the artifact upload. v4 names are immutable
  within a run and partial re-runs do not clear them, so re-running a
  publish job that failed at the deploy would have 409'd at the upload,
  before reaching the step that failed.
- Bound the repo1 poll by elapsed time rather than iteration count. With
  --max-time 30 on each request, 60 iterations of request+sleep could run
  for a full hour against a 45 minute job timeout, so a slow sync would
  be killed by the timeout and the actionable "re-run this job" message
  would never print.
- Fix the version guard. It rejected legal pre-release tags containing a
  hyphen (v2.1.0-alpha-1) and accepted both SNAPSHOT versions and leading
  zeros, where v02.1.0 would have irreversibly published a coordinate
  distinct from v2.1.0. Uses a portable case-insensitive SNAPSHOT match
  rather than ${var^^}, which needs bash 4 and so could not be exercised
  locally.
- Correct the byte-for-byte claim on the smoke test step. The agent jar
  and sources jar are identical across the two lifecycle passes because
  jar:jar and source:jar-no-fork skip as up-to-date, not because shade
  reads original-*.jar; the javadoc jar is regenerated and differs by its
  embedded timestamp.
- Add a concurrency group to java-agent.yml so superseded PR runs are
  cancelled, now that its trigger is unfiltered across a repo with many
  long-lived branches.

Signed-off-by: Shubham Jain <shubhamkjain@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant