feat(arc): GitOps-enforce arm64 node taint via Kyverno + ARC toleration

bootstrap/app-arc-arm-runners.yaml

@@ -17,7 +17,7 @@ spec:
     helm:
       releaseName: arc-arm-runners
       valueFiles:
-      - https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/arc/arm-scale-set-values.yaml?v=3
+      - https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/arc/arm-scale-set-values.yaml?v=4
   destination:
     server: https://kubernetes.default.svc
     namespace: arc-runners

bootstrap/appset-helm.yaml

@@ -88,7 +88,7 @@ spec:
         repoURL: https://kyverno.github.io/kyverno
         version: 3.5.0
         syncWave: '20'
-        valueURL: https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/kyverno/values.yaml?v=1
+        valueURL: https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/kyverno/values.yaml?v=2
       - app: loki
         chart: loki
         namespace: loki

docs/cluster.md

@@ -75,9 +75,13 @@ detailed in the findings sections of this document.
 | compute-lsis-2 | control-plane | amd64 | 81d | 10.2.248.31 | Ubuntu 24.04.4 / kernel 6.8.0-136-generic, containerd 2.3.4 | 32 CPU / 65.7 GiB / 110 pods (allocatable 29 CPU / 58.4 GiB) | 14% CPU / 53% mem |
 | lima-k3s-agent | worker | arm64 | 14h | 10.2.248.247 | kernel 6.8.0-138 | 8 CPU / 49.2 GiB / 110 pods | 2% CPU / 1% mem |
 
-Both nodes are Ready. `lima-k3s-agent` joined on 2026-09-07 and carries **no taints**
-— it is an active arm64 test node, not a cordoned spare. The control-plane node last
-restarted around 2026-09-01T12:53Z (Ready since that timestamp).
+Both nodes are Ready. `lima-k3s-agent` joined on 2026-09-07 and is an active arm64
+test node, not a cordoned spare. It carries a `kubernetes.io/arch=arm64:NoSchedule`
+taint, now **GitOps-enforced via Kyverno** (ClusterPolicy `node-arm64-arch-taint`,
+admission+background, in `kubernetes/arc/`) so it persists across kubelet
+re-provisioning; ARC ARM runners tolerate it (see `kubernetes/arc/README.md`).
+Resolved 2026-09-12. The control-plane node last restarted around 2026-09-01T12:53Z
+(Ready since that timestamp).
 
 ### K3s version
 
@@ -620,7 +624,7 @@ pulls (see §10); 159 VulnerabilityReports exist for scanned images.
 | 7 | Minor | Unmanaged namespace `reloader` | stakater reloader v1.4.21 (Deployment + ServiceAccount), created 2026-09-07T16:41Z, absent from git, no ArgoCD labels | Delete it or adopt it into GitOps |
 | 8 | Minor | Mutable image tags (5) | `synapse:latest`, `minio/minio:latest`, `link-shortener:latest` (live), `bash:latest` (init container), `minio/mc:latest` (one-shot) | Pin tags/digests for reproducibility |
 | 9 | Minor | Missing resource guarantees | gitea and oauth2-proxy pod specs have no CPU/memory requests or limits (others sampled have them) | Add requests/limits |
-| 10 | Minor | Untainted arm64 test node | `lima-k3s-agent` (joined 2026-09-07) carries no taints; unowned `arm-test` pod present | Taint with NoSchedule or decommission after testing |
+| 10 | Minor | Untainted arm64 test node — **resolved 2026-09-12** | `lima-k3s-agent` (joined 2026-09-07) carried no taints; unowned `arm-test` pod present | Tainted `kubernetes.io/arch=arm64:NoSchedule`, now GitOps-enforced via Kyverno `node-arm64-arch-taint` (admission+background) so it persists; ARC ARM runners tolerate it (`kubernetes/arc/`) |
 | 11 | Minor | Hubble flow buffer saturated | 4,095/4,095 at ~182 flows/s — flow visibility dropping events | Increase the flow-log queue or disable capture if unused |
 | 12 | Minor | Dangling registry secret | `ghcr-jbase-test` docker-registry secret in harbor ns, unreferenced | Delete if unused |
 | 13 | Minor | One historical backup failure | woodpecker-db 2026-09-02 "instance manager was restarted during backup" — recovered same day | Monitor; no action needed |

helm/releases/arc/arm-scale-set-values.yaml

@@ -23,6 +23,14 @@ template:
   spec:
     nodeSelector:
       kubernetes.io/arch: arm64
+    # Tolerate the arch taint (enforced by Kyverno node-arm64-arch-taint) so the
+    # arm runner pod can land on the tainted arm64 node. Listeners intentionally
+    # stay on the x64 node (no toleration added to listenerTemplate).
+    tolerations:
+      - key: "kubernetes.io/arch"
+        operator: "Equal"
+        value: "arm64"
+        effect: "NoSchedule"
     imagePullSecrets:
       - name: docker-hub-registry
       - name: harbor-registry-secret

helm/releases/kyverno/values.yaml

@@ -6,3 +6,31 @@ metricsPrometheusWidget:
   enabled: true
 reports:
   enableBroadcast: true
+
+# --- Kyverno engine config (resourceFilters) ----------------------------------
+# The node-arm64-arch-taint ClusterPolicy (kubernetes/arc/) needs the Kyverno
+# admission webhook to process Node objects so it can re-apply the
+# kubernetes.io/arch=arm64:NoSchedule taint on arm64 nodes. Kyverno's default
+# resourceFilters exclude `[Node,*,*]`, which would make the admission webhook
+# skip Nodes entirely. `resourceFiltersExclude` is a real chart hook (deep-merged
+# into the chart default) that removes exactly that one entry, keeping
+# `[Node/?*,*,*]` so Node *subresources* (status) stay filtered. excludeGroups
+# stays the chart default `system:nodes` (kubelet updates are covered by the
+# policy's background rescan instead).
+config:
+  resourceFiltersExclude:
+    - '[Node,*,*]'
+
+# --- One-off live-CM patch (post-merge, manual — NOT part of this diff) --------
+# The live `kyverno/kyverno` ConfigMap is annotated `helm.sh/resource-policy:
+# keep` and is NOT managed by ArgoCD/Helm, so the values change above only takes
+# effect on a fresh install / rebuild. To apply it to the running cluster, run
+# this single sanctioned data-level patch (read-only-verify the mutating webhook
+# catch-all on Node first: apiGroups `*`, resources `*/*`, ops CREATE/UPDATE on
+# `kyverno-policy-mutating-webhook-cfg`), substituting the exact INTENDED string:
+#
+#   kubectl patch configmap kyverno -n kyverno --type merge -p '{"data":{"resourceFilters":"<INTENDED STRING BELOW>"}}'
+#
+# INTENDED resourceFilters (current live value minus the standalone `[Node,*,*]`;
+# `[Node/?*,*,*]` kept), single line:
+#   [*/*,kyverno,*] [Event,*,*] [*/events,*,*] [APIService,*,*] [APIServiceGroup,*,*] [TokenReview,*,*] [SubjectAccessReview,*,*] [SelfSubjectAccessReview,*,*] [RuntimeClass,*,*] [ClusterRuntimeClass,*,*] [ConstrainedTemplatePolicy,*,*] [ClusterConstrainedTemplatePolicy,*,*] [ClusterPolicy,*,*] [ClusterPolicyException,*,*] [BackgroundScanReport,*,*] [ClusterBackgroundScanReport,*,*] [ClusterAdmissionReport,*,*] [AdmissionReport,*,*] [kyverno.io/*,*,*] [updaterequests,*,*] [kyverno.io/updaterequests,*,*] [namespaceinitializers,*,*] [namespaceinitializers.cert-manager.io,*,*] [Node/?*,*,*]

kubernetes/arc/README.md

@@ -87,6 +87,32 @@ Notes:
   `docker:28-dind` and the listener image are multiarch too — same image refs
   on both sets.
 
+## Node taint & toleration (arm64)
+
+The arm node `lima-k3s-agent` is tainted `kubernetes.io/arch=arm64:NoSchedule` so
+x64 work can never land on it (its dind has no binfmt). Two pieces keep this
+consistent under GitOps:
+
+- **Kyverno enforces the taint** — ClusterPolicy `node-arm64-arch-taint`
+  (`node-arm64-arch-taint.yaml`) runs admission+background and re-applies the
+  taint on any arm64 Node, so it survives kubelet re-provisioning (kubelet is in
+  `excludeGroups: system:nodes`, which the background rescan covers). Background
+  Node mutation is allowed by the `kyverno:update-nodes` ClusterRole
+  (`kyverno-node-mutation-rbac.yaml`), aggregated into the Kyverno background
+  controller via the `rbac.kyverno.io/aggregate-to-background-controller` label
+  (no separate binding). Kyverno's `resourceFilters` must not exclude
+  `[Node,*,*]` for the admission path to see Nodes — handled in
+  `helm/releases/kyverno/values.yaml` (`config.resourceFiltersExclude`) plus a
+  documented one-off patch to the live `keep`-annotated ConfigMap.
+- **ARM runners tolerate it** — `helm/releases/arc/arm-scale-set-values.yaml`
+  adds `template.spec.tolerations` for `kubernetes.io/arch=arm64:NoSchedule` so
+  the `ebpro-org-arm` runner pod can be scheduled onto the tainted node. Listeners
+  intentionally stay on the x64 node (`listenerTemplate` has no toleration).
+
+**VM-reboot behaviour:** `lima-k3s-agent` is a Lima dev VM. If it reboots, the arm
+runner pods vanish (and re-appear via `minRunners: 1` once the node returns), but
+the taint itself persists and is re-asserted by Kyverno — no manual re-taint needed.
+
 ## Ops notes
 
 - The 0.14.2 controller does **not** watch the `githubConfigSecret` change: after

kubernetes/arc/kustomization.yaml

@@ -6,3 +6,5 @@ resources:
   - crds/actions.github.com_autoscalingrunnersets.yaml
   - crds/actions.github.com_ephemeralrunners.yaml
   - crds/actions.github.com_ephemeralrunnersets.yaml
+  - node-arm64-arch-taint.yaml
+  - kyverno-node-mutation-rbac.yaml

kubernetes/arc/kyverno-node-mutation-rbac.yaml

@@ -0,0 +1,23 @@
+# RBAC for background Node mutation (see node-arm64-arch-taint.yaml).
+#
+# The Kyverno background controller rescan re-applies the arm64 node taint on
+# Node updates made by kubelet (which admission misses, as kubelet is in
+# `excludeGroups: system:nodes`). Background mutation requires the controller's
+# ServiceAccount to have update/patch on nodes.
+#
+# Rather than binding a dedicated ClusterRole, this role carries the Kyverno
+# aggregation label so the base `kyverno:background-controller` ClusterRole's
+# aggregationRule picks it up automatically — the existing
+# kyverno:background-controller ClusterRoleBinding (already created by the
+# Kyverno chart) then grants it. No extra binding is needed.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: kyverno:update-nodes
+  labels:
+    rbac.kyverno.io/aggregate-to-background-controller: "true"
+    argocd.argoproj.io/sync-wave: "-1"
+rules:
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["get", "list", "watch", "update", "patch"]

kubernetes/arc/node-arm64-arch-taint.yaml

@@ -0,0 +1,37 @@
+# Re-apply the arch taint on arm64 nodes so it survives kubelet re-provisioning.
+#
+# The ARM node (lima-k3s-agent) is intentionally tainted
+# kubernetes.io/arch=arm64:NoSchedule so x64 work never lands on it (its dind
+# has no binfmt). Kubelet periodically re-provisions Node objects and is in
+# Kyverno's `excludeGroups: system:nodes`, so admission alone would miss those
+# updates; this policy runs BOTH admission (real-time) and background (rescan)
+# to keep the taint persistent. Background Node mutation needs node update RBAC
+# — see kyverno-node-mutation-rbac.yaml (ClusterRole aggregated into the
+# Kyverno background controller).
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: node-arm64-arch-taint
+  annotations:
+    argocd.argoproj.io/sync-wave: "0"
+spec:
+  validationFailureAction: Audit
+  background: true
+  admission: true
+  emitWarning: false
+  rules:
+    - name: taint-arm64-nodes
+      match:
+        any:
+          - resources:
+              kinds: ["Node"]
+              selector:
+                matchLabels:
+                  kubernetes.io/arch: arm64
+      mutate:
+        patchStrategicMerge:
+          spec:
+            taints:
+              - key: kubernetes.io/arch
+                value: "arm64"
+                effect: NoSchedule