OCPBUGS-109633: on-prem: resign VIPs deterministically before node shutdown

[mkowalski]

Fixes OCPBUGS-109633: during systemctl reboot (e.g. an MCD-driven config-change reboot), keepalived non-deterministically loses the race against systemd's parallel teardown of container scopes and networking. When it loses, it is killed before sending the VRRP priority-0 resign advert and before removing the VIP from the interface. The stale VIP then keeps attracting new client connections which land directly on the local kube-apiserver — which is still serving its ~70s graceful drain with readyz=false — because the local haproxy VIP:6443→:9445 redirect is already gone. This is what the [Monitor:audit-log-analyzer] API LBs follow /readyz ... regression (Component Readiness regression 46508, triage 722) caught on vSphere upgrade jobs.

What this PR does

Adds a keepalived-vip-resign.service oneshot unit (rendered for on-prem platforms, enabled only with the OpenShift-managed LB and configured VIPs) whose ExecStop runs at shutdown before crio, kubelet and NetworkManager stop. The script:

1. No-ops unless the system is actually stopping (a plain unit stop/restart on a healthy node must not flap VIPs).
2. No-ops if no VIP is configured on this node.
3. Sends SIGTERM to keepalived (exact comm match pgrep -ox — an unanchored match would hit dynkeepalived or the script itself) so the clean resign (priority-0 advert + VIP removal) happens while the network is up, and waits up to 10s.
4. Fallback: if keepalived was not running or did not finish, kill it (pkill -9 -x, so it cannot re-add addresses) and remove the VIPs directly via ip addr del. VIP presence is checked with ip addr show to <vip>, i.e. kernel-side address comparison, so non-canonical IPv6 spellings still match.

The clean SIGTERM handoff is best-effort (transient crio-*.scope units are not ordered against this unit); the hard guarantee that no stale VIP survives into the drain window comes from the fallback.

Evidence from the failing runs

In job run 2086273547025518592: MCD reboots master-1 at 03:46:56; master-0 keepalived takes MASTER at 03:47:00 via master-down timeout (no Backup received priority 0 advertisement — contrast the clean handoff at 03:41:05 in the same run); peers' haproxy correctly marks master-1 DOWN at 03:47:11; yet flagged requests keep terminating on master-1's draining apiserver until 03:48:06 — only possible via a stale VIP on master-1. Same signature in runs 2086092373405732864 and 2085772778182545408.

Testing

• go test ./pkg/controller/template/... — includes a new table test covering unit enablement and rendered VIP list for: no VIPs, single stack, dual stack, ingress-only VIPs, and user-managed LB (disabled).
• Script logic exercised against a fake ip/pgrep/systemctl harness for: healthy-node stop (no-op), shutdown with dead keepalived (immediate force-removal, no wait), no VIP (no-op), deep-shutdown with systemctl unavailable (proceeds).
• Suggested cluster verification: on a VIP-holding master, loop systemctl reboot ~20x and confirm peers log Backup received priority 0 advertisement (or take over within ~4s) and ip addr on the rebooting node never retains the VIP into the apiserver drain; then rerun periodic-ci-openshift-release-main-ci-5.0-e2e-vsphere-ovn-upgrade.

Note for reviewers: this adds a new MachineConfig unit+file, so it rolls out with a drain+reboot of on-prem nodes, as usual for template changes.

Suggested follow-ups (from panel review, non-blocking)

These were raised during a multi-specialist review of this change and are deliberately left out of this PR to keep it scoped; filing them here so they are not lost:

1. Validate VIP strings at render time — the Go template interpolates PlatformStatus VIP strings into a root-executed shell script. The trust boundary is intact today (cluster-admin-controlled, API-validated fields), but a net.ParseIP check in the render path (or a template helper) would make the pattern fail closed if the source fields or their validation ever change.
2. Signal keepalived via its pod/pidfile instead of by process name — pgrep -ox keepalived is an exact-comm match, but name-based targeting is still spoofable by a local process named keepalived and has a theoretical pgrep→kill PID-reuse window. Resolving the PID via the static pod (crictl/conmon pidfile) would close both. Impact today is degraded-cleanliness only: the force-removal fallback still removes the VIPs.
3. Extend render-test platform matrix — the new test mutates the BareMetal platform only. Parameterizing over controller_config_{vsphere,openstack,nutanix,ovirt}.yaml (incl. the vSphere-UPI nil-PlatformStatus.VSphere case) would exercise every onPremPlatformAPIServerInternalIPs switch arm. Note controller_config_ovirt.yaml exists in test_data but is absent from the configs map in render_test.go, so the ovirt render path is currently untested by any unit test.
4. Shellcheck for embedded template scripts in CI — hack/verify-templates.sh does not lint the bash embedded in template yamls; a small extraction+shellcheck step would catch quoting/set -u regressions in this and sibling scripts (resolv-prepender.sh, mtu-migration.sh, ...).
5. e2e coverage — no existing e2e exercises VIP behavior across a node reboot. A metal-platform extended test that reboots the VIP-holding master and asserts the VIP is absent from that node during the apiserver drain would guard against regressions. Until then, the periodic-ci-openshift-release-main-ci-5.0-e2e-vsphere-ovn-upgrade monitor test is the effective signal.
6. Long-term home for this logic — the root cause is keepalived's shutdown handling losing the reboot race. This unit is the right near-term fix at the MCO layer, but the durable owner is arguably the keepalived static pod lifecycle in baremetal-runtimecfg (where VRRP state and VIP knowledge live). Cross-link from OCPBUGS-109633 so this workaround can be retired if/when that lands.

---

This PR was generated using AI. Please verify before acting on it.
Assisted-By: Claude Fable 5

Summary by CodeRabbit

• New Features
  • Added graceful shutdown handling for on-premises load balancers using virtual IPs.
  • Automatically removes virtual IPs during shutdown and force-cleans up when necessary.
  • Applies the shutdown behavior only to applicable managed load-balancer configurations.
• Tests
  • Added coverage for empty, single-stack, dual-stack, ingress-only, and user-managed configurations.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@mkowalski: This pull request references Jira Issue OCPBUGS-109633, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Fixes OCPBUGS-109633: during systemctl reboot (e.g. an MCD-driven config-change reboot), keepalived non-deterministically loses the race against systemd's parallel teardown of container scopes and networking. When it loses, it is killed before sending the VRRP priority-0 resign advert and before removing the VIP from the interface. The stale VIP then keeps attracting new client connections which land directly on the local kube-apiserver — which is still serving its ~70s graceful drain with readyz=false — because the local haproxy VIP:6443→:9445 redirect is already gone. This is what the [Monitor:audit-log-analyzer] API LBs follow /readyz ... regression (Component Readiness regression 46508, triage 722) caught on vSphere upgrade jobs.

What this PR does

Adds a keepalived-vip-resign.service oneshot unit (rendered for on-prem platforms, enabled only with the OpenShift-managed LB and configured VIPs) whose ExecStop runs at shutdown before crio, kubelet and NetworkManager stop. The script:

1. No-ops unless the system is actually stopping (a plain unit stop/restart on a healthy node must not flap VIPs).
2. No-ops if no VIP is configured on this node.
3. Sends SIGTERM to keepalived (exact comm match pgrep -ox — an unanchored match would hit dynkeepalived or the script itself) so the clean resign (priority-0 advert + VIP removal) happens while the network is up, and waits up to 10s.
4. Fallback: if keepalived was not running or did not finish, kill it (pkill -9 -x, so it cannot re-add addresses) and remove the VIPs directly via ip addr del. VIP presence is checked with ip addr show to <vip>, i.e. kernel-side address comparison, so non-canonical IPv6 spellings still match.

The clean SIGTERM handoff is best-effort (transient crio-*.scope units are not ordered against this unit); the hard guarantee that no stale VIP survives into the drain window comes from the fallback.

Evidence from the failing runs

In job run 2086273547025518592: MCD reboots master-1 at 03:46:56; master-0 keepalived takes MASTER at 03:47:00 via master-down timeout (no Backup received priority 0 advertisement — contrast the clean handoff at 03:41:05 in the same run); peers' haproxy correctly marks master-1 DOWN at 03:47:11; yet flagged requests keep terminating on master-1's draining apiserver until 03:48:06 — only possible via a stale VIP on master-1. Same signature in runs 2086092373405732864 and 2085772778182545408.

Testing

• go test ./pkg/controller/template/... — includes a new table test covering unit enablement and rendered VIP list for: no VIPs, single stack, dual stack, ingress-only VIPs, and user-managed LB (disabled).
• Script logic exercised against a fake ip/pgrep/systemctl harness for: healthy-node stop (no-op), shutdown with dead keepalived (immediate force-removal, no wait), no VIP (no-op), deep-shutdown with systemctl unavailable (proceeds).
• Suggested cluster verification: on a VIP-holding master, loop systemctl reboot ~20x and confirm peers log Backup received priority 0 advertisement (or take over within ~4s) and ip addr on the rebooting node never retains the VIP into the apiserver drain; then rerun periodic-ci-openshift-release-main-ci-5.0-e2e-vsphere-ovn-upgrade.

Note for reviewers: this adds a new MachineConfig unit+file, so it rolls out with a drain+reboot of on-prem nodes, as usual for template changes.

---

This PR was generated using AI. Please verify before acting on it.
Assisted-By: Claude Fable 5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0adf2583-b835-4901-ae33-774bb0e5ac01

📥 Commits

Reviewing files that changed from the base of the PR and between 0df05f6 and e689671.

📒 Files selected for processing (3)

• pkg/controller/template/render_keepalived_vip_resign_test.go
• templates/common/on-prem/files/keepalived-vip-resign-script.yaml
• templates/common/on-prem/units/keepalived-vip-resign.service.yaml

---

Walkthrough

The change adds a keepalived VIP resignation script and systemd unit for on-premises load balancers. A table-driven test validates rendering across VIP, stack, ingress, and load-balancer configurations.

Changes

Keepalived VIP resignation

Layer / File(s)	Summary
VIP resignation script templates/common/on-prem/files/keepalived-vip-resign-script.yaml	Adds shutdown-state checks, VIP detection, graceful keepalived termination, wait handling, and forced VIP cleanup.
Service wiring and rendering validation templates/common/on-prem/units/keepalived-vip-resign.service.yaml, pkg/controller/template/render_keepalived_vip_resign_test.go	Adds conditional systemd enablement, shutdown ordering, timeout protection, and rendering tests for master and worker configurations.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Mergeability Score: ⚪ Minimal · up to e6896

This change adds deterministic VIP resignation during node shutdown, with no actionable merge-blocking risk remaining beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant systemd
  participant keepalived-vip-resign.sh
  participant keepalived
  systemd->>keepalived-vip-resign.sh: Run resignation on shutdown
  keepalived-vip-resign.sh->>keepalived: Send SIGTERM
  keepalived-->>keepalived-vip-resign.sh: Remove configured VIPs
  keepalived-vip-resign.sh->>keepalived: Force cleanup if VIPs remain

Loading

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The added Go subtests use five literal static names, and the changed files contain no Ginkgo title calls or dynamic test-title construction.
Test Structure And Quality	✅ Passed	The added test uses standard testing.T, not Ginkgo. It creates no cluster resources and has no Eventually/Consistently waits; its fatal/error assertions include diagnostic messages.
Microshift Test Compatibility	✅ Passed	The only added test is a standard Go func Test... using testing.T; no Ginkgo e2e test, MicroShift API, or unsupported feature check applies.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The only added test is a standard Go testing.T rendering unit test, not a Ginkgo e2e test; it uses local configs and does not assume a multi-node cluster or require SNO guards.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff adds only a rendering test, an on-prem shell script, and a systemd unit; it introduces no pods, replicas, affinity, topology spread, node selectors, tolerations, or PDBs.
Ote Binary Stdout Contract	✅ Passed	The diff adds one ordinary Go test and YAML service/script; it changes no OTE entry point or process-level Go stdout, and shell echo output is not OTE binary stdout.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The added test is a standard Go testing.Test in pkg/controller/template, not a Ginkgo e2e test, and it performs no external connectivity; the IPv4 literals are outside this check's scope.
No-Weak-Crypto	✅ Passed	The PR diff adds a shutdown shell script, systemd unit, and rendering test; scans found no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret-token comparisons.
Container-Privileges	✅ Passed	The PR adds a host systemd unit and Ignition script, not container/Kubernetes manifests; added lines contain no privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, or allowPrivilegeEscalation set...
No-Sensitive-Data-In-Logs	✅ Passed	The added service logs operational state, a keepalived PID, interface, and configured VIP IPs; searches found no passwords, tokens, PII, session IDs, hostnames, or customer data.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the on-premises VIP resignation change and its purpose before node shutdown.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mkowalski
Once this PR has been reviewed and has the lgtm label, please assign yuqi-zhang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@mkowalski: This pull request references Jira Issue OCPBUGS-109633, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Fixes OCPBUGS-109633: during systemctl reboot (e.g. an MCD-driven config-change reboot), keepalived non-deterministically loses the race against systemd's parallel teardown of container scopes and networking. When it loses, it is killed before sending the VRRP priority-0 resign advert and before removing the VIP from the interface. The stale VIP then keeps attracting new client connections which land directly on the local kube-apiserver — which is still serving its ~70s graceful drain with readyz=false — because the local haproxy VIP:6443→:9445 redirect is already gone. This is what the [Monitor:audit-log-analyzer] API LBs follow /readyz ... regression (Component Readiness regression 46508, triage 722) caught on vSphere upgrade jobs.

What this PR does

Adds a keepalived-vip-resign.service oneshot unit (rendered for on-prem platforms, enabled only with the OpenShift-managed LB and configured VIPs) whose ExecStop runs at shutdown before crio, kubelet and NetworkManager stop. The script:

1. No-ops unless the system is actually stopping (a plain unit stop/restart on a healthy node must not flap VIPs).
2. No-ops if no VIP is configured on this node.
3. Sends SIGTERM to keepalived (exact comm match pgrep -ox — an unanchored match would hit dynkeepalived or the script itself) so the clean resign (priority-0 advert + VIP removal) happens while the network is up, and waits up to 10s.
4. Fallback: if keepalived was not running or did not finish, kill it (pkill -9 -x, so it cannot re-add addresses) and remove the VIPs directly via ip addr del. VIP presence is checked with ip addr show to <vip>, i.e. kernel-side address comparison, so non-canonical IPv6 spellings still match.

The clean SIGTERM handoff is best-effort (transient crio-*.scope units are not ordered against this unit); the hard guarantee that no stale VIP survives into the drain window comes from the fallback.

Evidence from the failing runs

In job run 2086273547025518592: MCD reboots master-1 at 03:46:56; master-0 keepalived takes MASTER at 03:47:00 via master-down timeout (no Backup received priority 0 advertisement — contrast the clean handoff at 03:41:05 in the same run); peers' haproxy correctly marks master-1 DOWN at 03:47:11; yet flagged requests keep terminating on master-1's draining apiserver until 03:48:06 — only possible via a stale VIP on master-1. Same signature in runs 2086092373405732864 and 2085772778182545408.

Testing

• go test ./pkg/controller/template/... — includes a new table test covering unit enablement and rendered VIP list for: no VIPs, single stack, dual stack, ingress-only VIPs, and user-managed LB (disabled).
• Script logic exercised against a fake ip/pgrep/systemctl harness for: healthy-node stop (no-op), shutdown with dead keepalived (immediate force-removal, no wait), no VIP (no-op), deep-shutdown with systemctl unavailable (proceeds).
• Suggested cluster verification: on a VIP-holding master, loop systemctl reboot ~20x and confirm peers log Backup received priority 0 advertisement (or take over within ~4s) and ip addr on the rebooting node never retains the VIP into the apiserver drain; then rerun periodic-ci-openshift-release-main-ci-5.0-e2e-vsphere-ovn-upgrade.

Note for reviewers: this adds a new MachineConfig unit+file, so it rolls out with a drain+reboot of on-prem nodes, as usual for template changes.

Suggested follow-ups (from panel review, non-blocking)

These were raised during a multi-specialist review of this change and are deliberately left out of this PR to keep it scoped; filing them here so they are not lost:

1. Validate VIP strings at render time — the Go template interpolates PlatformStatus VIP strings into a root-executed shell script. The trust boundary is intact today (cluster-admin-controlled, API-validated fields), but a net.ParseIP check in the render path (or a template helper) would make the pattern fail closed if the source fields or their validation ever change.
2. Signal keepalived via its pod/pidfile instead of by process name — pgrep -ox keepalived is an exact-comm match, but name-based targeting is still spoofable by a local process named keepalived and has a theoretical pgrep→kill PID-reuse window. Resolving the PID via the static pod (crictl/conmon pidfile) would close both. Impact today is degraded-cleanliness only: the force-removal fallback still removes the VIPs.
3. Extend render-test platform matrix — the new test mutates the BareMetal platform only. Parameterizing over controller_config_{vsphere,openstack,nutanix,ovirt}.yaml (incl. the vSphere-UPI nil-PlatformStatus.VSphere case) would exercise every onPremPlatformAPIServerInternalIPs switch arm. Note controller_config_ovirt.yaml exists in test_data but is absent from the configs map in render_test.go, so the ovirt render path is currently untested by any unit test.
4. Shellcheck for embedded template scripts in CI — hack/verify-templates.sh does not lint the bash embedded in template yamls; a small extraction+shellcheck step would catch quoting/set -u regressions in this and sibling scripts (resolv-prepender.sh, mtu-migration.sh, ...).
5. e2e coverage — no existing e2e exercises VIP behavior across a node reboot. A metal-platform extended test that reboots the VIP-holding master and asserts the VIP is absent from that node during the apiserver drain would guard against regressions. Until then, the periodic-ci-openshift-release-main-ci-5.0-e2e-vsphere-ovn-upgrade monitor test is the effective signal.
6. Long-term home for this logic — the root cause is keepalived's shutdown handling losing the reboot race. This unit is the right near-term fix at the MCO layer, but the durable owner is arguably the keepalived static pod lifecycle in baremetal-runtimecfg (where VRRP state and VIP knowledge live). Cross-link from OCPBUGS-109633 so this workaround can be retired if/when that lands.

---

This PR was generated using AI. Please verify before acting on it.
Assisted-By: Claude Fable 5

Summary by CodeRabbit

• New Features
• Added graceful shutdown handling for on-premises load balancers using virtual IPs.
• Automatically removes virtual IPs during shutdown and force-cleans up when necessary.
• Applies the shutdown behavior only to applicable managed load-balancer configurations.
• Tests
• Added coverage for empty, single-stack, dual-stack, ingress-only, and user-managed configurations.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[mkowalski]

/jira refresh

[openshift-ci-robot]

@mkowalski: This pull request references Jira Issue OCPBUGS-109633, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[mkowalski]

Follow-up 6 from the review section is now implemented: openshift/baremetal-runtimecfg#397 teaches dynkeepalived to resign the VIPs on host shutdown (clean stop via the control socket + bounded netlink force-removal, guarded so pod restarts on a healthy node are no-ops). The two changes are independent and idempotent together; once #397 proves out in CI, this unit can potentially be retired.

This comment was generated using AI. Please verify before acting on it.
Assisted-By: Claude Fable 5

[mkowalski]

/hold

Maybe openshift/baremetal-runtimecfg#397 is a better thing to do