From 7102df9922239a1e17a03475fdb8b844dc2097c7 Mon Sep 17 00:00:00 2001 From: Harold Sun Date: Sun, 13 Sep 2026 16:27:11 +0000 Subject: [PATCH 1/3] ci: drop the version-named SSM layer parameter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every merge to main has failed at deploy-beta since the v1.1.0 revert (2026-07-10) with: Resource of type 'AWS::SSM::Parameter' with identifier '/lambda-web-adapter/layer/x86_64/1.0.1' already exists. (AlreadyExists) The parameter's Name embedded ${CargoPkgVersion}, so a version change forces CloudFormation to replace the resource, and UpdateReplacePolicy: Retain left the old name behind un-managed. Releasing 1.1.0 orphaned /x86_64/1.0.1; reverting to 1.0.1 then tried to create that exact name and hit the orphan. Rollback restores the 1.1.0 name, so the next deploy repeats it — a permanent wedge, not a flake. It also took e2e-test-zip and e2e-test-oci down with it (both `needs: deploy-beta`), so main had no e2e coverage for two months. Removing the resource is the fix rather than dropping UpdateReplacePolicy: with only the current version ever tracked, the per-version parameter holds the same value as the /latest parameter beside it, so it earns nothing. The alternative — writing it with `aws ssm put-parameter --overwrite` from the workflow — needs ssm:PutParameter added to the PipelineExecutionRole in every beta, gamma, prod, and China account, which today grants only ssm:GetParameters on /lambda-web-adapter/e2e/*. Nothing reads a per-version parameter: the only consumer of either parameter is tests/e2e_tests/fixtures/go-httpbin-zip/template.yaml, which reads /lambda-web-adapter/layer/x86_64/latest. That one keeps its static name, so it updates in place and can never collide. No parameter is deleted by this change. DeletionPolicy: Retain applies when a resource is removed from the template, so the currently-tracked names are kept as the historical pointers, and the layer Description still records the version for `aws lambda list-layer-versions`. --- template-arm64.yaml | 13 ++++--------- template-x86_64.yaml | 17 ++++++++--------- 2 files changed, 12 insertions(+), 18 deletions(-) diff --git a/template-arm64.yaml b/template-arm64.yaml index 9b0af7b3..5f73ab07 100644 --- a/template-arm64.yaml +++ b/template-arm64.yaml @@ -39,15 +39,10 @@ Resources: Type: String Value: !Ref LambdaAdapterLayerArm64 - LambdaAdapterLayerArm64VersionParameter: - Type: AWS::SSM::Parameter - DeletionPolicy: Retain - UpdateReplacePolicy: Retain - Properties: - Name: !Sub '/lambda-web-adapter/layer/arm64/${CargoPkgVersion}' - Description: !Sub 'Layer ARN for the latest Lambda Web Adapter Arm64 Layer: ${CargoPkgVersion}' - Type: String - Value: !Ref LambdaAdapterLayerArm64 + # A per-version parameter (/lambda-web-adapter/layer/arm64/) used to + # live here. See the matching comment in template-x86_64.yaml for why it was removed: + # a Name change replaces the parameter, and UpdateReplacePolicy: Retain orphaned the + # old name, so re-deploying an already-used version failed with AlreadyExists. Outputs: LambdaAdapterLayerArm64Arn: diff --git a/template-x86_64.yaml b/template-x86_64.yaml index 2fa465ba..d857dbfc 100644 --- a/template-x86_64.yaml +++ b/template-x86_64.yaml @@ -39,15 +39,14 @@ Resources: Type: String Value: !Ref LambdaAdapterLayerX86 - LambdaAdapterLayerArm64VersionParameter: - Type: AWS::SSM::Parameter - DeletionPolicy: Retain - UpdateReplacePolicy: Retain - Properties: - Name: !Sub '/lambda-web-adapter/layer/x86_64/${CargoPkgVersion}' - Description: !Sub 'Layer ARN for the latest Lambda Web Adapter X86_64 Layer: ${CargoPkgVersion}' - Type: String - Value: !Ref LambdaAdapterLayerX86 + # A per-version parameter (/lambda-web-adapter/layer/x86_64/) used + # to live here. It was removed: because CloudFormation replaces a parameter when its + # Name changes, and UpdateReplacePolicy was Retain, every version bump orphaned the + # previous name. Deploying a version whose name had already been orphaned — after the + # v1.1.0 revert, or on any re-release — then failed with AlreadyExists and wedged + # every merge to main. Existing per-version parameters are retained, and the layer's + # Description still records the version, so `aws lambda list-layer-versions` maps a + # version to its layer ARN. Outputs: LambdaAdapterLayerX86Arn: From 7e16ee963d387c868dc6e44fe27e470b7455e488 Mon Sep 17 00:00:00 2001 From: Harold Sun Date: Sun, 13 Sep 2026 16:27:11 +0000 Subject: [PATCH 2/3] ci: retry tool downloads instead of piping curl into tar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit e2e-test-zip failed on the first green deploy-beta in two months, six steps before it ran anything: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ... curl: (35) Recv failure: Connection reset by peer tar: Error is not recoverable: exiting now Two problems. There was no retry, so one reset from get.nexte.st failed the job. And piping into tar discarded curl's exit status — the default shell for `run` is `bash -e`, without pipefail — so the diagnosis surfaced as a tar error rather than the network failure it was. Both downloads now retry and land in a file before extraction: cargo-nextest in all five places it is installed (merge test, merge e2e-test-zip, merge e2e-test-oci, pr, release) and mdBook in the docs workflow, which has the same pattern and would have been the next one to flake. --- .github/workflows/docs.yaml | 5 ++++- .github/workflows/merge.yaml | 24 +++++++++++++++++++++--- .github/workflows/pr.yaml | 8 +++++++- .github/workflows/release.yaml | 8 +++++++- 4 files changed, 39 insertions(+), 6 deletions(-) diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml index b1986f6c..9341e408 100644 --- a/.github/workflows/docs.yaml +++ b/.github/workflows/docs.yaml @@ -26,7 +26,10 @@ jobs: - name: Install mdBook run: | mkdir -p $HOME/bin - curl -sSL https://github.com/rust-lang/mdBook/releases/download/v0.4.40/mdbook-v0.4.40-x86_64-unknown-linux-gnu.tar.gz | tar -xz -C $HOME/bin + curl --retry 5 --retry-all-errors --retry-delay 5 -sSL \ + https://github.com/rust-lang/mdBook/releases/download/v0.4.40/mdbook-v0.4.40-x86_64-unknown-linux-gnu.tar.gz \ + -o /tmp/mdbook.tar.gz + tar -xz -C $HOME/bin -f /tmp/mdbook.tar.gz echo "$HOME/bin" >> $GITHUB_PATH - name: Build book diff --git a/.github/workflows/merge.yaml b/.github/workflows/merge.yaml index 4ccb85af..0bfda68b 100644 --- a/.github/workflows/merge.yaml +++ b/.github/workflows/merge.yaml @@ -36,7 +36,13 @@ jobs: uses: Swatinem/rust-cache@v2 - name: Install cargo-nextest - run: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin + # Retried, and downloaded to a file rather than piped into tar: the default + # shell has no pipefail, so curl's exit status was discarded and a transient + # connection reset surfaced as an unrecoverable tar error. + run: | + curl --retry 5 --retry-all-errors --retry-delay 5 -LsSf \ + https://get.nexte.st/latest/linux -o /tmp/cargo-nextest.tar.gz + tar zxf /tmp/cargo-nextest.tar.gz -C "${CARGO_HOME:-$HOME/.cargo}/bin" - name: linting run: | @@ -271,7 +277,13 @@ jobs: uses: Swatinem/rust-cache@v2 - name: Install cargo-nextest - run: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin + # Retried, and downloaded to a file rather than piped into tar: the default + # shell has no pipefail, so curl's exit status was discarded and a transient + # connection reset surfaced as an unrecoverable tar error. + run: | + curl --retry 5 --retry-all-errors --retry-delay 5 -LsSf \ + https://get.nexte.st/latest/linux -o /tmp/cargo-nextest.tar.gz + tar zxf /tmp/cargo-nextest.tar.gz -C "${CARGO_HOME:-$HOME/.cargo}/bin" - uses: actions/setup-python@v4 with: @@ -334,7 +346,13 @@ jobs: uses: Swatinem/rust-cache@v2 - name: Install cargo-nextest - run: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin + # Retried, and downloaded to a file rather than piped into tar: the default + # shell has no pipefail, so curl's exit status was discarded and a transient + # connection reset surfaced as an unrecoverable tar error. + run: | + curl --retry 5 --retry-all-errors --retry-delay 5 -LsSf \ + https://get.nexte.st/latest/linux -o /tmp/cargo-nextest.tar.gz + tar zxf /tmp/cargo-nextest.tar.gz -C "${CARGO_HOME:-$HOME/.cargo}/bin" - uses: actions/setup-python@v4 with: diff --git a/.github/workflows/pr.yaml b/.github/workflows/pr.yaml index cbe30b2c..fee6b066 100644 --- a/.github/workflows/pr.yaml +++ b/.github/workflows/pr.yaml @@ -28,7 +28,13 @@ jobs: uses: Swatinem/rust-cache@v2 - name: Install cargo-nextest - run: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin + # Retried, and downloaded to a file rather than piped into tar: the default + # shell has no pipefail, so curl's exit status was discarded and a transient + # connection reset surfaced as an unrecoverable tar error. + run: | + curl --retry 5 --retry-all-errors --retry-delay 5 -LsSf \ + https://get.nexte.st/latest/linux -o /tmp/cargo-nextest.tar.gz + tar zxf /tmp/cargo-nextest.tar.gz -C "${CARGO_HOME:-$HOME/.cargo}/bin" - name: linting run: | diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 4843a3d7..8ecef22a 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -32,7 +32,13 @@ jobs: uses: Swatinem/rust-cache@v2 - name: Install cargo-nextest - run: curl -LsSf https://get.nexte.st/latest/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin + # Retried, and downloaded to a file rather than piped into tar: the default + # shell has no pipefail, so curl's exit status was discarded and a transient + # connection reset surfaced as an unrecoverable tar error. + run: | + curl --retry 5 --retry-all-errors --retry-delay 5 -LsSf \ + https://get.nexte.st/latest/linux -o /tmp/cargo-nextest.tar.gz + tar zxf /tmp/cargo-nextest.tar.gz -C "${CARGO_HOME:-$HOME/.cargo}/bin" - name: linting run: | From 5633f7595b0ae7ffd842a2ab9779544bd50e1b30 Mon Sep 17 00:00:00 2001 From: Harold Sun Date: Sun, 13 Sep 2026 16:59:44 +0000 Subject: [PATCH 3/3] ci: fail the mdBook download on an HTTP error The mdBook step kept `-sSL` while the cargo-nextest steps use `-LsSf`, so it still had the failure mode this branch set out to remove. Without `--fail`, curl treats an HTTP error as a successful transfer: it writes the error page to /tmp/mdbook.tar.gz and exits 0, and `tar` then fails with "not in gzip format" instead of reporting the 403 or 404 that actually happened. `--retry` cannot help either, because curl never classifies the response as an error to retry. Reproduced against a non-existent release asset: without -f, curl exits 0 having written a 9-byte "Not Found" body and tar exits 2 with no usable message; with -f, curl exits 22 and says "The requested URL returned error: 404". The real asset still downloads and extracts (mdbook v0.4.40). Reported by aws-sam-tooling-bot on #843. --- .github/workflows/docs.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml index 9341e408..51084215 100644 --- a/.github/workflows/docs.yaml +++ b/.github/workflows/docs.yaml @@ -26,7 +26,11 @@ jobs: - name: Install mdBook run: | mkdir -p $HOME/bin - curl --retry 5 --retry-all-errors --retry-delay 5 -sSL \ + # -f so an HTTP error is an error: without it curl writes the error page to + # the output file and exits 0, and tar then fails with "not in gzip format" + # — the same misleading failure this commit set out to remove. It is also + # what lets --retry see a 5xx as retryable at all. + curl --retry 5 --retry-all-errors --retry-delay 5 -sSLf \ https://github.com/rust-lang/mdBook/releases/download/v0.4.40/mdbook-v0.4.40-x86_64-unknown-linux-gnu.tar.gz \ -o /tmp/mdbook.tar.gz tar -xz -C $HOME/bin -f /tmp/mdbook.tar.gz