ci: fix the deploy-beta SSM parameter collision and retry tool downloads - #843
Open
bnusunny wants to merge 3 commits into
Open
ci: fix the deploy-beta SSM parameter collision and retry tool downloads#843bnusunny wants to merge 3 commits into
bnusunny wants to merge 3 commits into
Conversation
Every merge to main has failed at deploy-beta since the v1.1.0 revert
(2026-07-10) with:
Resource of type 'AWS::SSM::Parameter' with identifier
'/lambda-web-adapter/layer/x86_64/1.0.1' already exists. (AlreadyExists)
The parameter's Name embedded ${CargoPkgVersion}, so a version change forces
CloudFormation to replace the resource, and UpdateReplacePolicy: Retain left
the old name behind un-managed. Releasing 1.1.0 orphaned /x86_64/1.0.1;
reverting to 1.0.1 then tried to create that exact name and hit the orphan.
Rollback restores the 1.1.0 name, so the next deploy repeats it — a permanent
wedge, not a flake. It also took e2e-test-zip and e2e-test-oci down with it
(both `needs: deploy-beta`), so main had no e2e coverage for two months.
Removing the resource is the fix rather than dropping UpdateReplacePolicy:
with only the current version ever tracked, the per-version parameter holds
the same value as the /latest parameter beside it, so it earns nothing. The
alternative — writing it with `aws ssm put-parameter --overwrite` from the
workflow — needs ssm:PutParameter added to the PipelineExecutionRole in every
beta, gamma, prod, and China account, which today grants only ssm:GetParameters
on /lambda-web-adapter/e2e/*.
Nothing reads a per-version parameter: the only consumer of either parameter is
tests/e2e_tests/fixtures/go-httpbin-zip/template.yaml, which reads
/lambda-web-adapter/layer/x86_64/latest. That one keeps its static name, so it
updates in place and can never collide.
No parameter is deleted by this change. DeletionPolicy: Retain applies when a
resource is removed from the template, so the currently-tracked names are kept
as the historical pointers, and the layer Description still records the version
for `aws lambda list-layer-versions`.
e2e-test-zip failed on the first green deploy-beta in two months, six steps before it ran anything: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ... curl: (35) Recv failure: Connection reset by peer tar: Error is not recoverable: exiting now Two problems. There was no retry, so one reset from get.nexte.st failed the job. And piping into tar discarded curl's exit status — the default shell for `run` is `bash -e`, without pipefail — so the diagnosis surfaced as a tar error rather than the network failure it was. Both downloads now retry and land in a file before extraction: cargo-nextest in all five places it is installed (merge test, merge e2e-test-zip, merge e2e-test-oci, pr, release) and mdBook in the docs workflow, which has the same pattern and would have been the next one to flake.
The mdBook step kept `-sSL` while the cargo-nextest steps use `-LsSf`, so it still had the failure mode this branch set out to remove. Without `--fail`, curl treats an HTTP error as a successful transfer: it writes the error page to /tmp/mdbook.tar.gz and exits 0, and `tar` then fails with "not in gzip format" instead of reporting the 403 or 404 that actually happened. `--retry` cannot help either, because curl never classifies the response as an error to retry. Reproduced against a non-existent release asset: without -f, curl exits 0 having written a 9-byte "Not Found" body and tar exits 2 with no usable message; with -f, curl exits 22 and says "The requested URL returned error: 404". The real asset still downloads and extracts (mdbook v0.4.40). Reported by aws-sam-tooling-bot on #843.
![aws-sam-tooling-bot[bot]](https://avatars.githubusercontent.com/b/1290?s=60&v=4){kind=small}
| # shell has no pipefail, so curl's exit status was discarded and a transient | ||
| # connection reset surfaced as an unrecoverable tar error. | ||
| run: | | ||
| curl --retry 5 --retry-all-errors --retry-delay 5 -LsSf \ |
There was a problem hiding this comment.
[GENERAL] The retry hardening covers connection resets and HTTP errors but not a stalled transfer: curl has no default overall or connect timeout, so a connection that opens and then hangs will block the step indefinitely. With --retry 5 each attempt can hang, which makes the worst case longer than before this change, not shorter. None of these jobs set timeout-minutes, so the fallback is the 6-hour GitHub job limit — the pipeline stays occupied and e2e-test-zip/e2e-test-oci stay queued behind it, the same blast radius this PR is trying to eliminate.
This repo already bounds curl elsewhere for exactly this reason — .github/scripts/verify-http.sh:21 uses -m 5 with the comment "so a hanging endpoint can't multiply the wait window."
Suggest adding per-attempt and total caps:
curl --retry 5 --retry-all-errors --retry-delay 5 \
--connect-timeout 10 --max-time 120 --retry-max-time 180 \
-LsSf https://get.nexte.st/latest/linux -o /tmp/cargo-nextest.tar.gz--max-time bounds each attempt (and is retried by --retry-all-errors as a transient failure), --retry-max-time bounds the whole sequence. The same gap applies to the three other copies of this step (.github/workflows/merge.yaml:277, .github/workflows/merge.yaml:346, .github/workflows/pr.yaml:31, .github/workflows/release.yaml:35) and to the mdBook download in .github/workflows/docs.yaml:31.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two failures seen on the last merge to main: a permanent
deploy-betawedge, and a flaky tool download that tooke2e-test-zipwith it.1.
deploy-betahas been wedged since the v1.1.0 revertEvery merge to main has failed since 2026-07-10 (run 29062863936, run 33441923283) with:
The parameter's
Nameembedded${CargoPkgVersion}, so changing the version forces CloudFormation to replace the resource, andUpdateReplacePolicy: Retainleft the old name behind un-managed:87f0c87) created/x86_64/1.1.0and orphaned/x86_64/1.0.1.3fc665d) tried to create that exact name → collided with the orphan.e2e-test-zipande2e-test-ociareneeds: [deploy-beta], so main had no e2e coverage for two months.Why removal rather than a different policy
UpdateReplacePolicy: Retainwould fix the collision, but then only the current version is ever tracked — and that parameter holds the identical value to the/latestparameter beside it. It earns nothing.aws ssm put-parameter --overwritefrom the workflow needsssm:PutParameteron thePipelineExecutionRolein every beta/gamma/prod/China account. Today that role grants onlyssm:GetParameterson/lambda-web-adapter/e2e/*, so the step would fail closed on first run.tests/e2e_tests/fixtures/go-httpbin-zip/template.yaml, which reads/lambda-web-adapter/layer/x86_64/latest. That name is static, so it updates in place and can never collide.No parameter is deleted by merging this.
DeletionPolicy: Retainapplies when a resource is removed from the template, so the currently tracked names are kept as historical pointers. The layerDescriptionstill records the version, soaws lambda list-layer-versionsmaps a version to its layer ARN.2. Tool downloads had no retry, and hid their own failure
On the first green
deploy-betain two months,e2e-test-zipdied at step 6 of 12 — beforesetup-sam, before assuming any role, before deploying or testing anything:Two problems: no retry, so a single reset from
get.nexte.stfails the job; and piping intotardiscards curl's exit status, because the default shell forrunisbash -ewithoutpipefail— so a network failure was reported as a tar error.Both downloads now retry and land in a file before extraction:
cargo-nextestin all five install sites (mergetest, mergee2e-test-zip, mergee2e-test-oci,pr,release) and mdBook indocs.yaml, which had the same pattern and would have been next to flake.Remediation already applied
The two orphaned parameters were deleted by hand in the beta account, which unwedged
deploy-betaon run 34767327920 (e2e-test-zippassed on re-run). Current beta state: all fourlambda-adapter-beta-*stacksUPDATE_COMPLETE,/x86_64/1.0.1re-created under stack management at layer:273,/latestadvanced to:273so it no longer advertises the backed-out v1.1.0 build.Gamma/prod/China accounts still hold the same class of orphan from earlier releases. They are harmless while version numbers are never reused, and this change stops new ones accruing — but a re-release of an existing version there would need the same one-time cleanup.
Validation
sam validate --lintclean on both templates.cargo-nextest 0.9.144.CargoPkgVersionis still referenced (the layerDescription), so the parameter is not orphaned in the template.