From ad2f15d92526fa7ff7be35e9cec6a58031644f46 Mon Sep 17 00:00:00 2001 From: Emmanuel Bruno Date: Sun, 13 Sep 2026 19:33:20 +0200 Subject: [PATCH] feat(arc): GitOps-enforce arm64 node taint via Kyverno + ARC toleration Enforce kubernetes.io/arch=arm64:NoSchedule on arm64 nodes (lima-k3s-agent) under GitOps so the taint persists across kubelet re-provisioning, and let the ARM runner pod tolerate it: - kubernetes/arc/node-arm64-arch-taint.yaml: Kyverno ClusterPolicy (admission+background) re-applying the taint on arm64 Nodes. - kubernetes/arc/kyverno-node-mutation-rbac.yaml: ClusterRole kyverno:update-nodes aggregated into the Kyverno background controller (rbac.kyverno.io/aggregate-to-background-controller label; no extra binding). - kubernetes/arc/kustomization.yaml: wire the two new resources. - helm/releases/arc/arm-scale-set-values.yaml: template.spec.tolerations for the arch taint (listeners stay on x64). - helm/releases/kyverno/values.yaml: config.resourceFiltersExclude ['[Node,*,*]'] so the admission webhook processes Node objects (keeps [Node/?*,*,*]). - bootstrap/app-arc-arm-runners.yaml: valueURL ?v=3 -> ?v=4. - bootstrap/appset-helm.yaml: kyverno valueURL ?v=1 -> ?v=2. - docs/cluster.md, kubernetes/arc/README.md: document the taint/toleration. Post-merge manual follow-up (NOT in this diff): the live kyverno/kyverno ConfigMap is keep-annotated and unmanaged, so apply the one-off data-level patch documented in helm/releases/kyverno/values.yaml to drop the standalone [Node,*,*] from the running cluster. --- bootstrap/app-arc-arm-runners.yaml | 2 +- bootstrap/appset-helm.yaml | 2 +- docs/cluster.md | 12 ++++-- helm/releases/arc/arm-scale-set-values.yaml | 8 ++++ helm/releases/kyverno/values.yaml | 28 ++++++++++++++ kubernetes/arc/README.md | 26 +++++++++++++ kubernetes/arc/kustomization.yaml | 2 + .../arc/kyverno-node-mutation-rbac.yaml | 23 ++++++++++++ kubernetes/arc/node-arm64-arch-taint.yaml | 37 +++++++++++++++++++ 9 files changed, 134 insertions(+), 6 deletions(-) create mode 100644 kubernetes/arc/kyverno-node-mutation-rbac.yaml create mode 100644 kubernetes/arc/node-arm64-arch-taint.yaml diff --git a/bootstrap/app-arc-arm-runners.yaml b/bootstrap/app-arc-arm-runners.yaml index 0003e168..1e8f90c8 100644 --- a/bootstrap/app-arc-arm-runners.yaml +++ b/bootstrap/app-arc-arm-runners.yaml @@ -17,7 +17,7 @@ spec: helm: releaseName: arc-arm-runners valueFiles: - - https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/arc/arm-scale-set-values.yaml?v=3 + - https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/arc/arm-scale-set-values.yaml?v=4 destination: server: https://kubernetes.default.svc namespace: arc-runners diff --git a/bootstrap/appset-helm.yaml b/bootstrap/appset-helm.yaml index 794e84a9..dd906378 100644 --- a/bootstrap/appset-helm.yaml +++ b/bootstrap/appset-helm.yaml @@ -88,7 +88,7 @@ spec: repoURL: https://kyverno.github.io/kyverno version: 3.5.0 syncWave: '20' - valueURL: https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/kyverno/values.yaml?v=1 + valueURL: https://raw.githubusercontent.com/ebpro/gitops/main/helm/releases/kyverno/values.yaml?v=2 - app: loki chart: loki namespace: loki diff --git a/docs/cluster.md b/docs/cluster.md index 7ca3b38e..a4aa92c5 100644 --- a/docs/cluster.md +++ b/docs/cluster.md @@ -75,9 +75,13 @@ detailed in the findings sections of this document. | compute-lsis-2 | control-plane | amd64 | 81d | 10.2.248.31 | Ubuntu 24.04.4 / kernel 6.8.0-136-generic, containerd 2.3.4 | 32 CPU / 65.7 GiB / 110 pods (allocatable 29 CPU / 58.4 GiB) | 14% CPU / 53% mem | | lima-k3s-agent | worker | arm64 | 14h | 10.2.248.247 | kernel 6.8.0-138 | 8 CPU / 49.2 GiB / 110 pods | 2% CPU / 1% mem | -Both nodes are Ready. `lima-k3s-agent` joined on 2026-09-07 and carries **no taints** -— it is an active arm64 test node, not a cordoned spare. The control-plane node last -restarted around 2026-09-01T12:53Z (Ready since that timestamp). +Both nodes are Ready. `lima-k3s-agent` joined on 2026-09-07 and is an active arm64 +test node, not a cordoned spare. It carries a `kubernetes.io/arch=arm64:NoSchedule` +taint, now **GitOps-enforced via Kyverno** (ClusterPolicy `node-arm64-arch-taint`, +admission+background, in `kubernetes/arc/`) so it persists across kubelet +re-provisioning; ARC ARM runners tolerate it (see `kubernetes/arc/README.md`). +Resolved 2026-09-12. The control-plane node last restarted around 2026-09-01T12:53Z +(Ready since that timestamp). ### K3s version @@ -620,7 +624,7 @@ pulls (see §10); 159 VulnerabilityReports exist for scanned images. | 7 | Minor | Unmanaged namespace `reloader` | stakater reloader v1.4.21 (Deployment + ServiceAccount), created 2026-09-07T16:41Z, absent from git, no ArgoCD labels | Delete it or adopt it into GitOps | | 8 | Minor | Mutable image tags (5) | `synapse:latest`, `minio/minio:latest`, `link-shortener:latest` (live), `bash:latest` (init container), `minio/mc:latest` (one-shot) | Pin tags/digests for reproducibility | | 9 | Minor | Missing resource guarantees | gitea and oauth2-proxy pod specs have no CPU/memory requests or limits (others sampled have them) | Add requests/limits | -| 10 | Minor | Untainted arm64 test node | `lima-k3s-agent` (joined 2026-09-07) carries no taints; unowned `arm-test` pod present | Taint with NoSchedule or decommission after testing | +| 10 | Minor | Untainted arm64 test node — **resolved 2026-09-12** | `lima-k3s-agent` (joined 2026-09-07) carried no taints; unowned `arm-test` pod present | Tainted `kubernetes.io/arch=arm64:NoSchedule`, now GitOps-enforced via Kyverno `node-arm64-arch-taint` (admission+background) so it persists; ARC ARM runners tolerate it (`kubernetes/arc/`) | | 11 | Minor | Hubble flow buffer saturated | 4,095/4,095 at ~182 flows/s — flow visibility dropping events | Increase the flow-log queue or disable capture if unused | | 12 | Minor | Dangling registry secret | `ghcr-jbase-test` docker-registry secret in harbor ns, unreferenced | Delete if unused | | 13 | Minor | One historical backup failure | woodpecker-db 2026-09-02 "instance manager was restarted during backup" — recovered same day | Monitor; no action needed | diff --git a/helm/releases/arc/arm-scale-set-values.yaml b/helm/releases/arc/arm-scale-set-values.yaml index 65888825..c6460be5 100644 --- a/helm/releases/arc/arm-scale-set-values.yaml +++ b/helm/releases/arc/arm-scale-set-values.yaml @@ -23,6 +23,14 @@ template: spec: nodeSelector: kubernetes.io/arch: arm64 + # Tolerate the arch taint (enforced by Kyverno node-arm64-arch-taint) so the + # arm runner pod can land on the tainted arm64 node. Listeners intentionally + # stay on the x64 node (no toleration added to listenerTemplate). + tolerations: + - key: "kubernetes.io/arch" + operator: "Equal" + value: "arm64" + effect: "NoSchedule" imagePullSecrets: - name: docker-hub-registry - name: harbor-registry-secret diff --git a/helm/releases/kyverno/values.yaml b/helm/releases/kyverno/values.yaml index e1310755..03a252a0 100644 --- a/helm/releases/kyverno/values.yaml +++ b/helm/releases/kyverno/values.yaml @@ -6,3 +6,31 @@ metricsPrometheusWidget: enabled: true reports: enableBroadcast: true + +# --- Kyverno engine config (resourceFilters) ---------------------------------- +# The node-arm64-arch-taint ClusterPolicy (kubernetes/arc/) needs the Kyverno +# admission webhook to process Node objects so it can re-apply the +# kubernetes.io/arch=arm64:NoSchedule taint on arm64 nodes. Kyverno's default +# resourceFilters exclude `[Node,*,*]`, which would make the admission webhook +# skip Nodes entirely. `resourceFiltersExclude` is a real chart hook (deep-merged +# into the chart default) that removes exactly that one entry, keeping +# `[Node/?*,*,*]` so Node *subresources* (status) stay filtered. excludeGroups +# stays the chart default `system:nodes` (kubelet updates are covered by the +# policy's background rescan instead). +config: + resourceFiltersExclude: + - '[Node,*,*]' + +# --- One-off live-CM patch (post-merge, manual — NOT part of this diff) -------- +# The live `kyverno/kyverno` ConfigMap is annotated `helm.sh/resource-policy: +# keep` and is NOT managed by ArgoCD/Helm, so the values change above only takes +# effect on a fresh install / rebuild. To apply it to the running cluster, run +# this single sanctioned data-level patch (read-only-verify the mutating webhook +# catch-all on Node first: apiGroups `*`, resources `*/*`, ops CREATE/UPDATE on +# `kyverno-policy-mutating-webhook-cfg`), substituting the exact INTENDED string: +# +# kubectl patch configmap kyverno -n kyverno --type merge -p '{"data":{"resourceFilters":""}}' +# +# INTENDED resourceFilters (current live value minus the standalone `[Node,*,*]`; +# `[Node/?*,*,*]` kept), single line: +# [*/*,kyverno,*] [Event,*,*] [*/events,*,*] [APIService,*,*] [APIServiceGroup,*,*] [TokenReview,*,*] [SubjectAccessReview,*,*] [SelfSubjectAccessReview,*,*] [RuntimeClass,*,*] [ClusterRuntimeClass,*,*] [ConstrainedTemplatePolicy,*,*] [ClusterConstrainedTemplatePolicy,*,*] [ClusterPolicy,*,*] [ClusterPolicyException,*,*] [BackgroundScanReport,*,*] [ClusterBackgroundScanReport,*,*] [ClusterAdmissionReport,*,*] [AdmissionReport,*,*] [kyverno.io/*,*,*] [updaterequests,*,*] [kyverno.io/updaterequests,*,*] [namespaceinitializers,*,*] [namespaceinitializers.cert-manager.io,*,*] [Node/?*,*,*] diff --git a/kubernetes/arc/README.md b/kubernetes/arc/README.md index 3f0f2208..37328126 100644 --- a/kubernetes/arc/README.md +++ b/kubernetes/arc/README.md @@ -87,6 +87,32 @@ Notes: `docker:28-dind` and the listener image are multiarch too — same image refs on both sets. +## Node taint & toleration (arm64) + +The arm node `lima-k3s-agent` is tainted `kubernetes.io/arch=arm64:NoSchedule` so +x64 work can never land on it (its dind has no binfmt). Two pieces keep this +consistent under GitOps: + +- **Kyverno enforces the taint** — ClusterPolicy `node-arm64-arch-taint` + (`node-arm64-arch-taint.yaml`) runs admission+background and re-applies the + taint on any arm64 Node, so it survives kubelet re-provisioning (kubelet is in + `excludeGroups: system:nodes`, which the background rescan covers). Background + Node mutation is allowed by the `kyverno:update-nodes` ClusterRole + (`kyverno-node-mutation-rbac.yaml`), aggregated into the Kyverno background + controller via the `rbac.kyverno.io/aggregate-to-background-controller` label + (no separate binding). Kyverno's `resourceFilters` must not exclude + `[Node,*,*]` for the admission path to see Nodes — handled in + `helm/releases/kyverno/values.yaml` (`config.resourceFiltersExclude`) plus a + documented one-off patch to the live `keep`-annotated ConfigMap. +- **ARM runners tolerate it** — `helm/releases/arc/arm-scale-set-values.yaml` + adds `template.spec.tolerations` for `kubernetes.io/arch=arm64:NoSchedule` so + the `ebpro-org-arm` runner pod can be scheduled onto the tainted node. Listeners + intentionally stay on the x64 node (`listenerTemplate` has no toleration). + +**VM-reboot behaviour:** `lima-k3s-agent` is a Lima dev VM. If it reboots, the arm +runner pods vanish (and re-appear via `minRunners: 1` once the node returns), but +the taint itself persists and is re-asserted by Kyverno — no manual re-taint needed. + ## Ops notes - The 0.14.2 controller does **not** watch the `githubConfigSecret` change: after diff --git a/kubernetes/arc/kustomization.yaml b/kubernetes/arc/kustomization.yaml index 5faa6291..7fc79683 100644 --- a/kubernetes/arc/kustomization.yaml +++ b/kubernetes/arc/kustomization.yaml @@ -6,3 +6,5 @@ resources: - crds/actions.github.com_autoscalingrunnersets.yaml - crds/actions.github.com_ephemeralrunners.yaml - crds/actions.github.com_ephemeralrunnersets.yaml + - node-arm64-arch-taint.yaml + - kyverno-node-mutation-rbac.yaml diff --git a/kubernetes/arc/kyverno-node-mutation-rbac.yaml b/kubernetes/arc/kyverno-node-mutation-rbac.yaml new file mode 100644 index 00000000..27df9266 --- /dev/null +++ b/kubernetes/arc/kyverno-node-mutation-rbac.yaml @@ -0,0 +1,23 @@ +# RBAC for background Node mutation (see node-arm64-arch-taint.yaml). +# +# The Kyverno background controller rescan re-applies the arm64 node taint on +# Node updates made by kubelet (which admission misses, as kubelet is in +# `excludeGroups: system:nodes`). Background mutation requires the controller's +# ServiceAccount to have update/patch on nodes. +# +# Rather than binding a dedicated ClusterRole, this role carries the Kyverno +# aggregation label so the base `kyverno:background-controller` ClusterRole's +# aggregationRule picks it up automatically — the existing +# kyverno:background-controller ClusterRoleBinding (already created by the +# Kyverno chart) then grants it. No extra binding is needed. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kyverno:update-nodes + labels: + rbac.kyverno.io/aggregate-to-background-controller: "true" + argocd.argoproj.io/sync-wave: "-1" +rules: + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch", "update", "patch"] diff --git a/kubernetes/arc/node-arm64-arch-taint.yaml b/kubernetes/arc/node-arm64-arch-taint.yaml new file mode 100644 index 00000000..bd306fe5 --- /dev/null +++ b/kubernetes/arc/node-arm64-arch-taint.yaml @@ -0,0 +1,37 @@ +# Re-apply the arch taint on arm64 nodes so it survives kubelet re-provisioning. +# +# The ARM node (lima-k3s-agent) is intentionally tainted +# kubernetes.io/arch=arm64:NoSchedule so x64 work never lands on it (its dind +# has no binfmt). Kubelet periodically re-provisions Node objects and is in +# Kyverno's `excludeGroups: system:nodes`, so admission alone would miss those +# updates; this policy runs BOTH admission (real-time) and background (rescan) +# to keep the taint persistent. Background Node mutation needs node update RBAC +# — see kyverno-node-mutation-rbac.yaml (ClusterRole aggregated into the +# Kyverno background controller). +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: node-arm64-arch-taint + annotations: + argocd.argoproj.io/sync-wave: "0" +spec: + validationFailureAction: Audit + background: true + admission: true + emitWarning: false + rules: + - name: taint-arm64-nodes + match: + any: + - resources: + kinds: ["Node"] + selector: + matchLabels: + kubernetes.io/arch: arm64 + mutate: + patchStrategicMerge: + spec: + taints: + - key: kubernetes.io/arch + value: "arm64" + effect: NoSchedule