ci: group Dependabot example updates, verify only what changed, auto-merge on green - #844
ci: group Dependabot example updates, verify only what changed, auto-merge on green#844bnusunny wants to merge 6 commits into
Conversation
There was no dependabot.yml, so Dependabot opened one pull request per advisory per manifest: 69 open at the time of writing, 13 of them against the single lockfile in examples/remix/remix-app and 8 against examples/remix-zip. Siblings in the same lockfile conflict as soon as one lands, which is why 13 of the 69 are already CONFLICTING. Each ecosystem present under examples/ now gets a group with `applies-to: security-updates`, so an example app is updated by one pull request instead of eight. `open-pull-requests-limit: 0` disables version updates and leaves security updates on, which preserves today's behavior: PRs for advisories only, not for every dependency that has drifted. The adapter's own Cargo.toml is deliberately absent — it ships as the lambda-adapter binary and its dependencies are reviewed by hand.
Verify Examples ran all 18 matrix entries for any change under examples/, so a lockfile bump in examples/remix/remix-app rebuilt and booted springboot, nextjs, deno-zip and the rest. At ~70 open Dependabot pull requests that is the dominant CI cost, and none of it is signal. A `select` job now diffs the pull request and emits one matrix per job kind, so that bump runs a single job. The example lists move to .github/example-matrix.json so the selector and the matrices share one source of truth. The selector fails safe — no base commit, a base commit that is not available locally, or a change to shared code (src/, layer/, Cargo.toml, the workflow itself, the matrix file) all verify everything. Pushes to main and manual runs are unaffected: they have no base commit and so verify everything. Adds `examples-verified`, one aggregate result for the whole workflow, treating `skipped` as a pass since that is what a filtered-out matrix means. Verified against real commits: a bump under examples/remix/remix-app selects only remix, the SnapStart merge (src/) selects everything, and a bump under an example with no matrix entry selects nothing.
Merges a Dependabot pull request once Verify Examples has gone green on it, if every file it changes is under examples/ and at least one build-and-boot job actually ran. Keyed off the completed workflow run rather than `gh pr merge --auto`: auto-merge is gated on the repository's *required* status checks, and Verify Examples is path-filtered to examples/**, so requiring its result would never report on a source-only pull request and would block it forever. The completed run is also tied to the head commit being merged, which addresses the stale verdict problem — PRs opened months ago still carry check results from the main of that day (#827 and older show validate:FAILURE for exactly that reason). Three guards, all necessary: * author is Dependabot; * every changed file is under examples/, checked against the PR's file list rather than its branch name, because grouped updates do not reliably encode the directory in the ref; * at least one test-* job succeeded, so an example with no matrix entry cannot ride in on a green run that only validated templates. Scope is deliberate: examples are demo apps, where a bad bump costs a broken sample. The adapter's own dependencies, the workflows, and the layer templates stay manual.
Records the two behaviors this config depends on, both now confirmed rather than assumed: Grouping is per directory. PRs #804 and #811 carry the identical update set (body-parser + express) and even the identical branch hash multi-be700a2db9, yet Dependabot raised them as two separate PRs, one per directory. Cross-directory batching requires `group-by: dependency-name`, which applies to version updates only. So this config yields one PR per example app per ecosystem. `open-pull-requests-limit: 0` stops version updates without stopping security updates: those are exempt from the limit and do not count toward it.
![aws-sam-tooling-bot[bot]](https://avatars.githubusercontent.com/b/1290?s=60&v=4){kind=small}
Three fixes from review on #844, all in the merge step. Pin the merge to the verified commit. The evidence came from a workflow run tied to workflow_run.head_sha, but `gh pr merge` merged whatever the head was when the API call ran. Dependabot force-pushes its branches on rebase and recreate, so the head can move while the run finishes, and the result would be an unverified commit squashed into main. The step now compares the current head to the verified one and skips if it moved, and passes --match-head-commit to close the remaining window. Make the scope guard fail closed. `outside=$(gh api ... | grep -v '^examples/' || true)` applied `|| true` to the whole pipeline, so a rate-limited or failed API call left `outside` empty and the pull request read as example-only — the one check keeping the adapter's Cargo.toml, the workflows, and the layer templates out of auto-merge. The API call is now separate from the filtering, with an explicit refusal on an empty list. Confirmed by simulation: the old form merges on an API failure, the new form does not. Gate on coverage of the changed set, not a job count. "At least one test-* job succeeded" was weaker than its comment claimed: select-examples.sh silently drops changed examples with no matrix entry, so a pull request touching one covered and one uncovered example passed while the second was never built or booted. Every changed example must now appear in .github/example-matrix.json, read at the verified commit. Not reachable with today's config — all 69 open Dependabot pull requests touch exactly one example, and grouping is per directory — but it becomes reachable the moment grouping spans directories, and the job count is the wrong thing to assert either way. All six guard paths exercised against a stubbed gh: merge, moved head, files outside examples/, uncovered example, covered-plus-uncovered, and no test jobs.
|
|
||
| permissions: | ||
| contents: write | ||
| pull-requests: write |
There was a problem hiding this comment.
[BUG] The permissions block grants contents: write and pull-requests: write, but the final gate calls the Actions API:
verified=$(gh api "repos/$REPO/actions/runs/$RUN_ID/jobs" --paginate \
-q '[.jobs[] | select(.name | startswith("test-")) | select(.conclusion == "success")] | length')Listing jobs for a workflow run requires Actions read access, and because this workflow declares an explicit permissions block, every scope not listed is set to none. The call returns 403, and since the step runs under set -euo pipefail, the failing command substitution aborts the step — so the workflow goes red on every Dependabot PR and nothing ever merges. It fails closed, which is the right direction, but the automation is inert.
permissions:
actions: read
contents: write
pull-requests: writeSecondary issue on the same call: gh api --paginate applies -q per page, so once a run exceeds one page of jobs the output is one count per page (e.g. 25\n3). [[ "$verified" -eq 0 ]] then hits an arithmetic syntax error, which evaluates as false and lets the merge proceed. Today's maximum is ~22 jobs so it stays single-page, but the guard silently inverts if the matrix grows. Summing instead of counting per page is robust regardless of pagination:
verified=$(gh api "repos/$REPO/actions/runs/$RUN_ID/jobs" --paginate \
-q '.jobs[] | select(.name | startswith("test-")) | select(.conclusion == "success") | .name' | wc -l)| examples-cargo: | ||
| applies-to: security-updates | ||
| patterns: | ||
| - "*" |
There was a problem hiding this comment.
[GENERAL] The six ecosystems configured (npm, pip, gomod, maven, nuget, cargo) miss bundler: examples/sinatra/app/src/Gemfile and Gemfile.lock are in the tree. Those manifests keep the pre-PR behavior — ungrouped, and without commit-message.prefix, so Dependabot writes Bump rack from 2.2.3 to 2.2.4.
That matters beyond grouping: commitlint.yaml runs on every pull request with no path filter, and commitlint.config.js enforces type-enum with a 120-char header. A default Dependabot message has no conventional type, so those PRs land with a red Commit Lint check and need a manual retitle — exactly the toil the chore prefix removes for the other six ecosystems.
- package-ecosystem: bundler
directories:
- "/examples/**"
schedule:
interval: weekly
open-pull-requests-limit: 0
commit-message:
prefix: chore
groups:
examples-bundler:
applies-to: security-updates
patterns:
- "*"The three findings from the earlier review are addressed in this revision, so I did not re-raise them: the merge is now pinned via the head_now comparison plus --match-head-commit, the gh api file listing is out of the || true pipeline with an explicit empty-list refusal, and the per-example coverage check against example-matrix.json closes the "one test-* job succeeded" gap.
Three fixes from the second review on #844. Add `actions: read`. Declaring a permissions block sets every unnamed scope to none, and the coverage gate lists the triggering run's jobs, so the Actions API would have returned 403 and aborted the step on every Dependabot pull request. It fails closed, but the automation would never have merged anything. commitlint-comment.yaml declares the same scope for the same reason. Count job names instead of asking jq for a length. `--paginate` applies `-q` per page, so a run spanning two pages yields one count per line ("18\n4"), and `[[ "18\n4" -eq 0 ]]` is an arithmetic syntax error that evaluates false — skipping the refusal and merging. Verified in bash: the multi-line form errors with "syntax error in expression" and takes the else branch. Single-page today at 22 jobs, but the failure direction is fail-open, and the headroom is smaller than it looks. Add the bundler ecosystem for examples/sinatra/app/src/Gemfile, which the original sweep missed. Beyond grouping, this is what gives those PRs a conventional commit prefix: Commit Lint runs on every pull request with no path filter, and #799 shows what the default message costs — commit "bump com.fasterxml.jackson.core:jackson-databind", Lint Commit Messages red.
|
|
||
| # Shared inputs every example is built against: the adapter itself, the layer | ||
| # wrapper, this workflow's own machinery. | ||
| if grep -qE '^(src/|layer/|Cargo\.toml$|Cargo\.lock$|\.github/workflows/examples\.yaml$|\.github/scripts/|\.github/example-matrix\.json$)' <<<"$changed"; then |
There was a problem hiding this comment.
[BUG] Three of the "shared path" alternatives in this guard are unreachable, because examples.yaml's own trigger filter never lets the workflow run for them:
on:
pull_request:
branches: [main]
paths:
- "examples/**"
push:
branches: [main]
paths:
- "src/**"A pull request that touches only .github/example-matrix.json, .github/scripts/select-examples.sh, or .github/workflows/examples.yaml does not match examples/**, so Verify Examples never starts and the emit_all branch below is never reached. Same for src/, layer/, Cargo.toml and Cargo.lock — those only reach this script when the PR also changes something under examples/.
This matters more after this PR than before it, because the matrix definition and the selector script are now the single source of truth for what gets verified, and neither is covered by CI. A malformed example-matrix.json (invalid JSON, a wrong expect_body, a missing port) or a regression in this script merges unverified and then breaks the next unrelated examples PR, where the cause is no longer obvious. This PR is itself an instance: it rewires all three matrices and Verify Examples will not run on it.
Adding the paths this script already treats as shared to the trigger filter makes the fail-safe branch reachable and self-verifying:
on:
pull_request:
branches: [main]
paths:
- "examples/**"
- "src/**"
- "layer/**"
- "Cargo.toml"
- "Cargo.lock"
- ".github/workflows/examples.yaml"
- ".github/scripts/**"
- ".github/example-matrix.json"| # --match-head-commit closes the remaining window: if the branch moves | ||
| # between the check above and this call, the merge is refused rather than | ||
| # applied to an unverified commit. | ||
| gh pr merge "$PR_NUMBER" --repo "$REPO" --squash --delete-branch \ |
There was a problem hiding this comment.
[GENERAL] The merge decision consults only the Verify Examples run; no other check on the pull request is examined. commitlint.yaml triggers on every pull_request to main with no paths filter, and its final step exits 1 on a lint failure, so a Dependabot PR can be sitting with a red Commit Lint check and still be squash-merged here — nothing in the gate looks at it. The bundler comment in .github/dependabot.yml cites #799 as an existing instance of exactly that check going red on a Dependabot PR.
prefix: chore should make the common case pass, but the gate has no way to notice when it does not, and any check added to this repository later is silently ignored too. Since the workflow already has the PR number, one more query closes it:
# Every other check on the PR must be conclusive, not just Verify Examples.
bad=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json statusCheckRollup \
-q '.statusCheckRollup[]
| select((.conclusion // "") | IN("SUCCESS","SKIPPED","NEUTRAL") | not)
| .name // .context')
if [[ -n "$bad" ]]; then
echo "PR #$PR_NUMBER has checks that are not green; refusing to merge:"
echo "$bad" | sed 's/^/ /'
exit 0
fiNote that statusCheckRollup reflects the PR's current head — worth running it after the existing head_now comparison so a moved branch is still reported as "moved" rather than "not green".
| # which has no conventional type, and Commit Lint runs on every pull request with no | ||
| # path filter. #799 is the evidence — its commit message is | ||
| # "bump com.fasterxml.jackson.core:jackson-databind" and its Commit Lint check is red. | ||
| - package-ecosystem: bundler |
There was a problem hiding this comment.
[GENERAL] The bundler entry was added because a security-update PR for an ecosystem with no matching updates entry gets Dependabot's default commit message, which has no conventional type and turns Commit Lint red. That reasoning applies unchanged to github-actions, which is in the dependency graph for this repository (every workflow here pins actions such as actions/checkout@v4, actions/upload-artifact@v4, Swatinem/rust-cache@v2, orhun/git-cliff-action@v4) and does receive advisories — so an alert produces an ungrouped PR titled bump actions/... from X to Y, failing Commit Lint and needing a manual amend before it can land.
It is outside examples/, so the auto-merge scope guard will correctly refuse it either way; the cost is the same manual fixup the bundler entry was added to avoid. An entry with the limit still at 0 changes nothing about which PRs are opened, only how they are named:
- package-ecosystem: github-actions
directories:
- "/"
schedule:
interval: weekly
open-pull-requests-limit: 0
commit-message:
prefix: ci
groups:
github-actions:
applies-to: security-updates
patterns:
- "*"1 participant
Automates the Dependabot example PRs end to end: fewer PRs, verification that only runs what changed, and merge on green.
Today: 69 open Dependabot PRs, 13 already
CONFLICTING, each triggering all 18 Verify Examples matrix entries, and nodependabot.ymlat all.1. Group the updates —
.github/dependabot.ymlWithout a config, Dependabot opens one PR per advisory per manifest: 13 against
examples/remix/remix-app's single lockfile, 8 againstexamples/remix-zip. Siblings in one lockfile conflict as soon as any of them lands, which is where the 13CONFLICTINGcome from.Each ecosystem under
examples/(npm, pip, gomod, maven, nuget, cargo) now has a group withapplies-to: security-updates— plaingroupsonly batches version updates, so that key is load-bearing.open-pull-requests-limit: 0disables version updates and leaves security updates on, preserving today's behavior rather than unleashing a flood of routine bumps. Flip it per ecosystem if you want those later. The adapter's ownCargo.tomlis deliberately unmanaged.2. Verify only what changed —
selectjob +.github/example-matrix.jsonA lockfile bump in
examples/remix/remix-appcurrently rebuilds and bootsspringboot,nextjs,deno-zipand the rest. At ~70 open PRs that cost dominates CI and produces no signal.selectdiffs the PR and emits one matrix per job kind. The example lists move to.github/example-matrix.jsonso the selector and the matrices share one source of truth.It fails safe — no base commit, an unavailable base commit, or a change to shared code (
src/,layer/,Cargo.toml, the workflow, the matrix file) all verify everything. Pushes to main and manual runs have no base commit, so they verify everything as before.Verified against real history:
3a21552examples/remix/remix-app/package-lock.jsonimage=[remix], zip/stream empty4c38d8fsrc/**34d3a29examples/nextjs-response-streaming/**3. Merge on green —
dependabot-automerge.yamlKeyed off the completed Verify Examples run, not
gh pr merge --auto. Auto-merge is gated on the repo's required status checks, and Verify Examples is path-filtered toexamples/**— so requiring its result would never report on a source-only PR and would block it forever. The completed run is also tied to the head commit being merged, which handles the stale-verdict problem: #827 and older still showvalidate:FAILUREfrom amainof months ago and have never re-run.Three guards:
examples/— checked against the PR's file list, not its branch name, because grouped updates don't reliably encode the directory in the ref.test-*job succeeded, so an example with no matrix entry can't ride in on a run that only validated templates.Validated against live data: a real Verify Examples run reports 18 successful
test-*jobs; #842 (Dependabot, examples-only) passes the scope check; #843 (this repo's CI fix) is correctly rejected on.github/workflows/*.What this does and does not auto-merge
Guard 3 splits the current backlog roughly in half:
remix13,remix-zip8,nextjs5,fastapi-background-tasks4,gin2,fastapi1,fastapi-zip1,expressjs1)datadog7,nextjs-response-streaming5,sveltekit-ssr-zip4,datadog-zip4,nextjs-zip3, others)The obvious follow-up is adding matrix entries for the busiest uncovered examples —
datadog,nextjs-response-streaming,nextjs-zip,sveltekit-ssr-zip— which converts them to auto-merge without touching this machinery. Note that "no matrix entry" is not a coverage regression; those examples were never boot-tested, they were just sitting in runs that booted other examples.Draining the 69
Land this, then
@dependabot closethe per-advisory PRs and let the grouped ones regenerate against currentmain. One fresh, verified PR per example app beats rebasing 13 competing patches against one lockfile.Validation
bash -nclean on the selector, the auto-merge script, and the gate script.success skipped skipped, exits 1 onsuccess failure.Grouping semantics, confirmed
The open question from the first revision — whether a grouped PR covers one directory or spans them — is settled, and the config needs no change.
Dependabot groups per directory. The proof is in this repo's own PRs: #804 and #811 carry the identical update set (
body-parser+express) and even the identical branch hashmulti-be700a2db9, yet were raised as two separate PRs, one per directory. Every multi-dependency PR checked (#821, #818, #811, #804, #741) touches exactly one directory, with that directory embedded in the ref ahead of themulti-<hash>suffix. Cross-directory batching requiresgroup-by: dependency-name, which is version-updates-only and so cannot apply to these security updates.open-pull-requests-limit: 0is likewise confirmed safe: security update PRs "are not subject to this limit and do not count toward it", so version-update noise stays off while advisory fixes keep flowing.Both behaviors are now recorded in
.github/dependabot.ymlso the next reader does not have to re-derive them.