feat(arc): GitOps-enforce arm64 node taint via Kyverno + ARC toleration - #8
Merged
Merged
Conversation
Enforce kubernetes.io/arch=arm64:NoSchedule on arm64 nodes (lima-k3s-agent) under GitOps so the taint persists across kubelet re-provisioning, and let the ARM runner pod tolerate it: - kubernetes/arc/node-arm64-arch-taint.yaml: Kyverno ClusterPolicy (admission+background) re-applying the taint on arm64 Nodes. - kubernetes/arc/kyverno-node-mutation-rbac.yaml: ClusterRole kyverno:update-nodes aggregated into the Kyverno background controller (rbac.kyverno.io/aggregate-to-background-controller label; no extra binding). - kubernetes/arc/kustomization.yaml: wire the two new resources. - helm/releases/arc/arm-scale-set-values.yaml: template.spec.tolerations for the arch taint (listeners stay on x64). - helm/releases/kyverno/values.yaml: config.resourceFiltersExclude ['[Node,*,*]'] so the admission webhook processes Node objects (keeps [Node/?*,*,*]). - bootstrap/app-arc-arm-runners.yaml: valueURL ?v=3 -> ?v=4. - bootstrap/appset-helm.yaml: kyverno valueURL ?v=1 -> ?v=2. - docs/cluster.md, kubernetes/arc/README.md: document the taint/toleration. Post-merge manual follow-up (NOT in this diff): the live kyverno/kyverno ConfigMap is keep-annotated and unmanaged, so apply the one-off data-level patch documented in helm/releases/kyverno/values.yaml to drop the standalone [Node,*,*] from the running cluster.
There was a problem hiding this comment.
🟡 Changes recommended
The Kyverno policy does not restore the taint during background scans, and the sync-wave is in labels instead of annotations.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds GitOps-managed Kyverno taint enforcement for the arm64 node and an ARC runner toleration.
Changes:
- Adds Kyverno policy and RBAC for arm64 node tainting.
- Adds ARC ARM runner toleration.
- Updates Kyverno values, bootstrap references, and documentation.
File summaries
| File | Summary |
|---|---|
kubernetes/arc/README.md |
Documents arm64 taint and toleration behavior. |
kubernetes/arc/node-arm64-arch-taint.yaml |
Adds the arm64 node mutation policy. Critical (3 votes): background scanning will not restore the taint; use mutate-existing or another reconciler. |
kubernetes/arc/kyverno-node-mutation-rbac.yaml |
Adds Kyverno Node mutation permissions. Moderate (2 votes): sync-wave is under labels instead of annotations. |
kubernetes/arc/kustomization.yaml |
Includes the new resources. |
helm/releases/kyverno/values.yaml |
Adjusts Kyverno Node resource filters. |
helm/releases/arc/arm-scale-set-values.yaml |
Adds the ARM runner toleration. |
docs/cluster.md |
Updates node inventory and finding status. |
bootstrap/appset-helm.yaml |
Updates the Kyverno values cache version. |
bootstrap/app-arc-arm-runners.yaml |
Updates the ARC values cache version. |
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| selector: | ||
| matchLabels: | ||
| kubernetes.io/arch: arm64 | ||
| mutate: |
Comment on lines
+17
to
+19
| labels: | ||
| rbac.kyverno.io/aggregate-to-background-controller: "true" | ||
| argocd.argoproj.io/sync-wave: "-1" |
emmanuelbruno
added a commit
that referenced
this pull request
Sep 14, 2026
…state consistency Follow-up to #8 (Copilot review on PR #8). - fix(arc): argocd.argoproj.io/sync-wave must be an annotation, not a label. The label value '-1' is rejected by K8s label validation, which blocked creation of the kyverno:update-nodes ClusterRole and put the whole arc ArgoCD app into SyncError. - docs(cluster): align stale 'untainted' arm64 node statements (capacity section, finding #5 evidence) with the GitOps-enforced taint. Assessment of #8 Copilot points: - critical (background scan not restoring taint): false positive for Kyverno 1.15 (background: true re-applies mutation to existing resources; mutateExisting defaults true since v1.12). The live resourceFilters Node exclusion is addressed by the values resourceFiltersExclude hook plus the sanctioned one-off patch to the keep-annotated ConfigMap. - moderate (sync-wave under labels): valid, fixed in this commit. Pending fresh Copilot re-review did not land within the polling window; only delta since last Copilot review is the docs fix addressing its own nit.
emmanuelbruno
added a commit
that referenced
this pull request
Sep 15, 2026
…lt drift + correct kyverno CM comment (#17) Close-out for the merged ARC/Kyverno PR #8. The Kyverno clusterpolicies.kyverno.io CRD sets rule-level skipBackgroundRequests to default: true. The arc policy's manifest omitted the field, so the API server defaulted it to true live -> the arc app stayed permanently OutOfSync AND the background rescan was disabled (defeating the policy's purpose of re-asserting the arm64 taint after node re-provisioning). ArgoCD SSA cannot clear a defaulted field, so only an explicit value in git fixes it durably. - kubernetes/arc/node-arm64-arch-taint.yaml: add skipBackgroundRequests: false to the taint-arm64-nodes rule. - helm/releases/kyverno/values.yaml: correct the stale comment block. The kyverno ConfigMap IS ArgoCD-managed (ArgoCD owns its data keys; the keep annotation only prevents deletion, not update), and the resourceFiltersExclude hook already drops [Node,*,*] at render time, so NO manual one-off ConfigMap patch is required. Remove the stale/wrong literal resourceFilters string.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enforces the
kubernetes.io/arch=arm64:NoScheduletaint on arm64 nodes (lima-k3s-agent) under GitOps via Kyverno so the taint persists across kubelet re-provisioning, and lets the ARC ARM runner pod tolerate it.Closes finding #10 (untainted arm64 test node) from
docs/cluster.md.Changes
kubernetes/arc/node-arm64-arch-taint.yamlClusterPolicy(admission+background) re-applying the arch taint on arm64 Nodeskubernetes/arc/kyverno-node-mutation-rbac.yamlClusterRole kyverno:update-nodes, aggregated into the Kyverno background controller via therbac.kyverno.io/aggregate-to-background-controllerlabel (no extra binding needed)kubernetes/arc/kustomization.yamlhelm/releases/arc/arm-scale-set-values.yamltemplate.spec.tolerationsfor the arch taint (listeners intentionally stay on x64)helm/releases/kyverno/values.yamlconfig.resourceFiltersExclude: ['[Node,*,*]']so the admission webhook processes Node objects (keeps[Node/?*,*,*]); documents the one-off live-CM patchbootstrap/app-arc-arm-runners.yaml?v=3→?v=4bootstrap/appset-helm.yaml?v=1→?v=2docs/cluster.mdkubernetes/arc/README.mdWhy both admission + background
Kubelet is in Kyverno's
excludeGroups: system:nodes, so admission alone would miss kubelet-driven Node updates. The policy runs admission (real-time) and background (rescan) to keep the taint persistent; background Node mutation is authorized by the aggregatedkyverno:update-nodesClusterRole.Validation (offline)
kubectl kustomize kubernetes/arc→ renders cleanly (EXIT 0, no errors); both new resources present and correct.helm template kyverno …/values.yaml→[Node,*,*]removed,[Node/?*,*,*]kept,system:nodeskept. Diff vs. the pure chart default confirms the exclude surgically removes only the standalone[Node,*,*].The live
kyverno/kyvernoConfigMap is annotatedhelm.sh/resource-policy: keepand is not managed by ArgoCD/Helm, so thevalues.yamlchange only takes effect on a fresh install/rebuild. After merge + sync, apply the single sanctioned data-level patch documented inhelm/releases/kyverno/values.yamlto drop the standalone[Node,*,*]from the running cluster (the exact INTENDEDresourceFiltersstring is in that comment block):Post-merge verification checklist
arc-arm-runnersapp Synced/Healthy after?v=4kyvernoapp picks up?v=2values (fresh render only — live CM needs the manual patch)ClusterPolicy node-arm64-arch-taintisReady(background + admission enabled)kyverno:update-nodesaggregated intokyverno:background-controller(verify the role includes nodeupdate/patch)kubernetes.io/arch=arm64:NoScheduletoleration and schedules onlima-k3s-agentresourceFiltersno longer contains the standalone[Node,*,*]