scaling down will send unnecessary alerts

[BraceCY]

The npd component checks the health status of kube-proxy via the /healthz interface

node-problem-detector/pkg/healthchecker/types/types.go

Line 83 in 2baac6c

kubeProxyHealthCheckEndpoint = fmt.Sprintf("http://%s/healthz", net.JoinHostPort(hostAddress, kubeProxyPort))

When cluster-autoscaler performs elastic scaling down, it will add the ToBeDeletedByClusterAutoscaler taint to nodes. When a node has the ToBeDeletedByClusterAutoscaler taint, the /healthz interface of kube-proxy will consider itself to be in an unhealthy state.
https://github.com/kubernetes/kubernetes/blob/d4672d9a9db5687c4561048c4840ef1f8ccb3c75/pkg/proxy/healthcheck/proxy_health.go#L188

The elastic scaling down of cluster-autoscaler is standard behavior and should not trigger any alarms, as it can cause confusion.

[BraceCY]

/kind bug

[hakman]

@BraceCY thanks for reporting this.
Why do you think it's a NPD issue and not a kube-proxy issue?
CC @wangzhen127 @SergeyKanzhelev

[BraceCY]

@hakman thank you for your reply!

The Kubernetes SIG-Network has discussed the behavior of kube-proxy at kubernetes/enhancements#3836.
I believe that KEP-3836 explains the scenario.

Goals:
Offer a better capability for connection draining when terminating nodes, especially for load balancers that support this.

Kube-proxy provides two interfaces: /healthz and /livez. The /healthz interface checks node eligibility, excluding nodes that are being deleted or tainted with "ToBeDeletedTaint", among other conditions. The /livez interface only checks the status of kube-proxy.

I think it is reasonable for nodes to be in a state of being deleted or tainted with "ToBeDeletedTaint" as part of normal operations. Therefore, it would be more appropriate for NPD to send alarms based on the /livez interface.

CC @wangzhen127 @SergeyKanzhelev

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[hakman]

/remove-lifecycle rotten

[dpacgdm]

Opened a fix in #1306 — switch the kube-proxy probe from /healthz to /livez so Cluster Autoscaler scale-down no longer triggers false unhealthy alerts.