Drain traffic from instances that are no longer serving - #253
Merged
Conversation
A networking NetworkService needs to know whether a member can serve, but the only health signal on a NetworkInterface was Programmed, written solely by the VPC fabric. Where the fabric is not attached nothing ever writes it, so every member reads unhealthy and no traffic is served even though membership and locality resolve correctly. Compute holds the interface and knows whether the instance behind it is serving, so compute publishes that. A HolderAvailable condition is projected onto every NetworkInterface an instance holds, mirroring the instance's own Available condition. Networking reads it and never learns that the holder is a workload. Anything other than Available=True projects as False carrying the instance's own reason, so Unknown drains the member rather than leaving it silently taking traffic. The projection runs on every reconcile, not only on a transition, so interfaces that predate it converge without their workload being redeployed, and it writes only when the condition moves. Key changes: - Project HolderAvailable onto each interface named by status.networkInterfaces[].networkInterfaceRef - Run the projection on the suspension path too, so a suspended instance drains before the hub is told - Grant the controller networkinterfaces/status write access
Deletion is the drain path that matters. Scale-down, rolling update, and redeploy all end an instance by deleting it, and the deletion reconcile returns before availability was ever projected. Nothing clears the instance's Available condition on the way out either, so HolderAvailable stayed True for the whole deletion window and the edge kept routing to a process shutting down. Deletion is now judged ahead of the instance's own Available condition, and the deletion reconcile drains the interfaces it held before releasing anything else. That write is best effort: an instance that cannot reach its interfaces must still finish deleting, or a transient API error strands it in Terminating forever. A retained interface outlives the instance that held it, so a True left behind could be inherited by the next instance in the slot before it is serving. Draining on delete removes it at the source, and the projection derives the condition wholly from the current holder rather than merging with what is on the interface, so a successor starts False regardless. The projection also moved above the quota and status-update returns in the main reconcile. Those returns skipped it, and an instance stuck waiting on quota is exactly one that should not be receiving traffic. Key changes: - Report HolderTerminating=False for an instance with a deletion timestamp - Drain from reconcileDeletion before quota cleanup and finalizer removal, logging rather than returning a drain failure - Project before the early returns that previously skipped it
scotwells
marked this pull request as ready for review
August 26, 2026 20:45
scotwells
enabled auto-merge
August 26, 2026 21:19
privateip
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A NetworkService only sends traffic to instances that are actually serving. Compute is the one service that knows whether an instance is serving, so it now says so directly on the network interface each instance holds.
Networking deliberately does not know what a workload is. It reads a single condition,
HolderAvailable, meaning "whatever holds this interface reports itself ready to serve". This PR makes compute the writer of that condition, sourced from the instance's ownAvailablestate.What consumers get
An instance starts serving and its addresses begin receiving traffic. An instance stops serving, for any reason, and traffic stops reaching it before it goes away.
That second half is the part that matters in practice. Scaling down, rolling out a new version, or redeploying all delete instances, and traffic now drains from an instance as soon as it begins terminating rather than continuing until it disappears. Instances that never became available, including ones held up waiting on quota, never receive traffic at all.
Notes for reviewers
Draining happens before any other deletion work, and it is best effort: a failed status write can never keep an instance stuck in Terminating.
Availability is derived fresh from the current holder on every pass rather than merged with whatever is already on the interface. With
reclaimPolicy: Retainan interface outlives its instance so the next instance in the slot reuses the same addresses, and deriving fresh is what stops a new instance inheriting its predecessor's word before it is serving.Pairs with datum-cloud/network-services-operator#411, which reads this condition. The two sides currently agree by string rather than by symbol, because compute pins a network-services-operator version predating the constant. Worth swapping to the imported symbol at the next pin bump.