Network services: load balance across the locations an application runs in - #411
Draft
scotwells wants to merge 21 commits into
Draft
Network services: load balance across the locations an application runs in#411scotwells wants to merge 21 commits into
scotwells wants to merge 21 commits into
Conversation
A NetworkService is a named set of endpoints spanning every location a consumer runs in. Members are selected by label against network interface claims, and a proxy names the service as its backend. Each edge ranks the service's locations by distance from itself and serves from the best one, so consumers configure no routing. This lands the consumer-facing type only. Controllers, webhooks, and RBAC follow separately. Key changes: - Add api/v1alpha/networkservice_types.go with spec.networkInterfaceClaims (label selector), spec.ports, and spec.trafficDistribution - Report per-location member, healthy, and serving counts in status, with a summary rollup backing the print columns - Define Ready, MembersResolved, and EndpointsReachable conditions plus the reasons that explain a service not serving - Constrain the selector to set matchLabels or matchExpressions, so an empty selector cannot make every claim in the namespace a member - Enforce unique port names and port numbers within a service - Keep protocol and trafficDistribution.strategy single-valued enums so later values are additive - Regenerate CRDs, deepcopy, and API reference docs
A NetworkService names a label selector and the platform has to turn it into the endpoint set behind it. The membership controller resolves the selector against the network interface claims in the service's own namespace, rolls the matches up per location, and reports what it found so an over-broad or stale selector is visible instead of silent. Membership is not the same question as health. A matched claim is a member; it is healthy only once it reports Programmed, so a location scaling up shows members it is not yet serving from rather than looking broken. A location leaves rotation when none of its members are healthy, which is what moves traffic down each edge's ranking. One network per service cannot be expressed in CEL, because it is a fact about the set of matched claims rather than about the service. The controller detects it and refuses to narrow the membership silently. Key changes: - Add NetworkServiceReconciler, watching claims and re-enqueuing every service whose selector reaches the claim that changed - Report status.locations, status.summary, and the MembersResolved, EndpointsReachable and Ready conditions - Count a claim carrying no location label as a member and name the discrepancy on MembersResolved rather than dropping it - Register the controller with the manager and add its RBAC
A rule can now name a NetworkService and one of its ports as its backend, alongside the endpoint, connector, and instance forms. The controller resolves the service's membership and publishes it as EndpointSlices, so a proxy follows instances appearing, disappearing, and moving between locations without an edit. Membership is resolved by evaluating the service's claim selector directly rather than reading a status a sibling controller has not written yet. Each member's address is taken from the claim's external addresses first, falling back to its interface addresses: backhaul from the edge crosses the public internet, so an externally reachable address is what the edge needs. Key changes: - Add networkService to HTTPProxyRuleBackend, referencing a service by name and a port by name so the reference survives a port change, and extend the struct-level CEL to keep it exclusive with endpoint, connector, and instance - Set each endpoint's zone from the claim's location label, which is what Envoy Gateway turns into a locality, and its ready condition from whether the claim reports Programmed - Shard members across EndpointSlices at the 100-endpoint limit, labelling every shard with the service name so the set stays discoverable as one service's endpoints - Report NetworkServiceBackendNotFound when the service or port is missing, and NetworkServiceMembersUnreferenced when members were published that a single backendRef cannot reach - Watch NetworkService and NetworkInterfaceClaim so membership changes reprogram the proxies that consume them
The NetworkService type, its membership reconciler, and the networkService HTTPProxy backend had only unit tests. This runs them against the real two-cluster environment and holds what it found. The scenario selects three network interface claims across two locations, reports two of them programmed, and requires the resulting endpoints to reach the running proxy: one Envoy locality per location, the members' real IPAM addresses inside them, the unprogrammed member carried but out of rotation. A real request then has to come back 200 from a member, and the unprogrammed member has to have taken none of it. Key changes: - Add test/e2e-edge/networkservice-endpoints, covering the data-plane path, a selector spanning two networks (MultipleNetworks), and an unprogrammed claim counting as a member but not as healthy - Add a host-network stand-in for a member so the edge talks to a real server at the address IPAM allocated, since this environment runs no compute - Run the scenario in the default e2e set
A NetworkService resolves its members by reading networking.datumapis.com/location off the NetworkInterfaceClaim, but nothing ever wrote that key onto a claim: only the projected NetworkInterface carried it. Every member was therefore unlocated — counted in the summary, absent from status.locations, and emitted to Envoy with no zone, leaving one empty locality and nothing for nearest-location routing to rank. The cell already knows the location it serves, so the claim reconciler records it on the claim it fulfills, next to the labels compute puts there. Both travel the same path to any published copy. Key changes: - Label the claim from the resolved cell location before binding, with an additive merge patch so other writers' labels survive and a claim already naming the location is left alone - Drop the hand-applied labels from the networkservice-endpoints e2e scenario and assert the operator's instead; the cell serves one location, so the scenario now proves one locality reaches Envoy and the multi-location rollup stays with the envtest suite
NetworkService had no IAM definition, so it could not be granted to a user the way every other networking type can. Add the ProtectedResource that registers the type and its permissions with the IAM system, and surface those permissions through the existing networking roles. The type is namespaced, consumer-authored and a child of a Project, so it takes the same CRUD permission set as the other consumer-authored networking types. Status is written by the controller under its own RBAC, so no status permission is exposed. Key changes: - Add config/iam/protected-resources/networkservices.yaml declaring networkservices under networking.datumapis.com with a Project parent - Grant read to the Network Viewer role and write to the Network Admin role, matching how network interface claims are handled
A NetworkService resolved its members from NetworkInterfaceClaims, which are never published to a consumer's project and whose Programmed condition has no writer anywhere: NSO seeds it Unknown and only the interface ever carries the data plane's verdict. A service therefore selected objects a consumer cannot see and judged health from a field nothing sets. Interfaces are projected, already carry the location, and carry the conditions, so membership now selects them. Two things a label selector cannot say stay in the controller. A retained interface outlives its claim with every label intact, so only one in the Bound phase is a member; healthy is the interface reporting Programmed. Compute's keys reach the interface by an allow-list at bind, so a consumer can still select a whole workload without a key they control landing on one the platform reads. Key changes: - Rename spec.networkInterfaceClaims to spec.networkInterfaces and the wrapper type with it; NoMatchingClaims becomes NoMatchingInterfaces - Resolve membership through one helper both the service controller and the proxy's endpoint synthesis use, so the two cannot diverge on which interfaces count; endpoints read spec.addresses and spec.externalAddresses, external first as before - Carry the claim's compute.datumapis.com keys and the cell's location onto the interface it binds, replacing the propagated set rather than merging it, and carry the same keys onto the published copy - Fix writeProjection, which merged labels and so kept a held-by or location key forever after its source dropped it; a stale label defeats the retired-capacity check - Drop the location stamp on the claim, which nothing reads now
Every NetworkService member in staging is unhealthy and no service can go Ready. Health read the interface's Programmed condition, whose only writer is the VPC fabric's BGP advertisement controller, gated on a fabric attachment. Staging has none, so Programmed is permanently Unknown and every member with it. Health now reads HolderAvailable: whatever holds the interface reporting itself available to serve. The holder writes it, the networking API never does and never learns what a holder is. Programmed keeps its meaning and its seam. It is the fabric-realization signal, and it is still what would say an address is actually carried. It no longer gates service health, so a service can now report healthy while its members' addresses are unroutable. Key changes: - add the HolderAvailable condition and its reason vocabulary to NetworkInterface, in the default condition set and seeded Unknown by the claim reconciler alongside Prepared and Programmed - read it from isServiceMemberHealthy, which both the service controller and the HTTPProxy endpoint builder already go through, so location health and EndpointSlice ready/serving follow one signal - cover the flip in envtest, and cover the condition reaching the consumer's project copy across both projection hops
A retained interface outlives its holder by design: the claim is named after the slot, so the next instance filling that slot rebinds the same interface and returns to the same addresses. Nothing cleared HolderAvailable when the holder let go, and seedCondition leaves an existing condition alone, so the rebind inherited the previous holder's True. The new holder was therefore counted healthy from the instant it bound, and the edge sent it a service's traffic before it had said anything. markInterfaceAvailable is the one place an interface becomes unheld, so both Retain paths are closed by resetting there. Key changes: - reset HolderAvailable to Unknown/HolderReleased when a retained interface is unbound, so the next holder starts silent - state the rule on the condition: NSO only ever writes Unknown, seeding it and taking it back; True and False are the holder's alone - cover the release-and-rebind window, which no test reached
A holder watches the claim it owns, not the interface behind it, so a reset written on a path with no claim event reaches nothing and the holder sits on Unknown until an unrelated cause wakes it. Both callers are claim releases, so the reset converges today, and nothing in this file says why that is load-bearing.
A networkService backend is always reached over plaintext HTTP: the synthesized EndpointSlice hardcodes an http appProtocol, and the BackendTLSPolicy the Gateway controller writes is keyed on an https one. Setting tls.hostname on such a backend therefore did nothing at all, and the consumer had no way to learn that. Reject the combination at admission instead, alongside the CEL rule that already governs which backend kinds may be combined. Key changes: - Add a CEL rule on HTTPProxyRuleBackend refusing tls with networkService - Say so in the field's own documentation - Pin both the rejection and the still-valid endpoint+tls case against a real apiserver in test/crd
A NetworkService that resolves no members still gets an EndpointSlice, carrying no endpoints. That reads like litter, but the rule's backendRef names it: the Gateway controller resolves an EndpointSlice backendRef by getting the upstream object, and its caller abandons the entire route loop on the first error. Withholding the slice would therefore stop every other HTTPProxy on the same Gateway from being reprogrammed for as long as the service had no members, rather than degrading just that rule. Leave the behaviour alone and make it expensive to remove by accident. Key changes: - Record the reason, and the fan-out caveat that would invalidate it, at the branch that writes the empty slice - Pin that an empty slice still resolves into a downstream Service, and that a withheld one surfaces as NotFound out of the route loop - Strengthen the httpproxy-side case to assert the backendRef and port the empty slice has to keep
The manager and the cell mount karmada-hub-kubeconfig with optional set, so kubelet gives them an empty directory when the secret is not there yet and they crashloop on a missing file until the next volume sync swaps in the real contents. It clears itself after a couple of restarts, which is exactly the kind of thing that flakes in CI. Ordering is not the lever: karmada-hub-secret already runs ahead of link-clusters and the crashloop happens anyway, because the empty mount is established long before either. Neither process can federate without the file, so it was never optional. Drop the flag and let the pods wait in ContainerCreating for the mount, the way the downstream and IPAM kubeconfigs in the same overlay already do; wait-ready still gates on the result. Key changes: - Drop optional from both karmada-hub-kubeconfig volumes in config/e2e - Say in the up sequence why the secret lands where it does
The condition was set to Unknown on every reconcile and never to anything else. Nothing reported request outcomes, so it could not become True or False, and MembersUnreachable was defined and read but never written. A condition that cannot change states implies a capability the platform does not have. Reachability is judged by the edge, which already ejects members that stop answering, and reporting that to the control plane per request would scale with traffic rather than with topology. Key changes: - Remove EndpointsReachable and MembersUnreachable - Ready now summarizes membership and serving locations only
An edge cannot reach a VPC pod today. galactic-cni publishes a per-pod EndpointSlice carrying the pod's address and the SRv6 SID that galactic-vrf turns into an egress route, but it writes it in the cell and nothing carries it anywhere else. A member holds a ULA that is unroutable from an edge, so a request against one times out on connect. This adds the cell-to-hub hop. Every slice galactic labels with galactic.datum.net/tenant-id is copied to the federation hub, where the propagation policy already selecting EndpointSlices carries it to every gateway cluster with no Karmada change. The copy is transport, not interpretation: address type, endpoints, conditions, ports and both galactic annotations cross unread. Nothing here parses the SID, because galactic-vrf refuses a slice this operator has normalised and would say nothing about why. Key changes: - Add VPCEndpointSliceWriteBackReconciler, modelled on the interface write-back: same hub client, same project routing, same periodic sweep - Name a copy after the location that published it, never after the pod. A cell is also a gateway cluster, so propagation returns the copy to the namespace the original sits in; sharing the name would hand galactic's own object to federation and resurrect it after the pod goes - Drop the pod owner reference, which names nothing on the hub - Publish on observation only. A failed read, a slice carrying no endpoint, and another location's copies all leave what is already up alone
What allocates a network's identity on the fabric runs centrally and has no multi-cluster runtime, so it could not watch a Network where one lives. It was driven by NetworkContexts instead, which meant it could never tell a network that had been deleted from one that is required nowhere right now, and it withdrew the identity in both cases. The Network now joins the set already mirrored onto the hub, so that component reads the network itself. Key changes: - Add Network to the replicator's default resource set, beside the NetworkContext and Subnet it already carries - Skip the downstream-to-upstream status sync, as a Network's status is written by NSO's own controllers - Propagate no labels: the copy is read on the hub and deliberately goes no further, where a context carries a location label only so a per-location policy can select it onto a cell - Install the Network CRD outside the project control plane, since the hub now stores a copy
Envoy programmed a NetworkService's members correctly but could not reach any of them: it egresses in the default network namespace, where no route to the tenant ULA exists, so every connection timed out without ever establishing. Two defects kept the socket bind from working. The extension server only recorded a VRF binding for instance backends, which name a galactic EndpointSlice directly. A networkService backend names no slice, so it was skipped entirely. Its tenant is now joined by member address instead: the addresses the backend forwards to, looked up against the tenant label galactic puts on the slice owning each address. Those slices reach an edge either from the local CNI or as federated copies, so a member resolves wherever it runs. The VRF device name was a placeholder that matched nothing galactic creates. It is now byte-identical to intf.GenerateInterfaceNameVRF, and keyed on the VPC alone rather than on vpc-plus-attachment, because galactic shares one device across every attachment of a VPC on a node. The instance path had the same wrong name, so it can never have bound a real device either. Key changes: - Join a networkService backend to its tenant by member address, over a single pass of the cluster's EndpointSlices - Canonicalise addresses on both sides so the join survives two components spelling one IPv6 address differently - Leave the binding unresolved when members span more than one VPC; a socket bind is a property of the whole cluster, and either choice would black-hole the other VPC's members - Skip the bind rather than emit a name no device answers to - Register discovery/v1 in the server test scheme, which the production scheme already carries through client-go
An edge reached a member at its external address when it had one, on the reasoning that backhaul crosses the public internet. It does not: an edge reaches a member over the fabric, and the address it dials is the tenant one. Dialling an external address leaves the fabric and crosses the public internet to reach an origin meant to stay private. Key changes: - Publish only the addresses a member holds inside its network - Assert an external address is never published
A NetworkService with no serving members answered slowly and generically. Envoy's default 50% panic threshold meant that below half healthy it ignored health entirely and forwarded to members whose holder had explicitly said they were not serving, burning the full 10s connect timeout on each attempt before returning a raw 503. Panic mode exists because active health checking can be wrong at scale. NetworkService health is not probed but declared — compute writes HolderAvailable from the instance's own state — so zero healthy is ground truth, not a measurement artifact. Turning the threshold off makes Envoy fail immediately and report no healthy upstream, which is a condition specific enough to brand; with panic mode on it reports a connect failure or timeout instead, indistinguishable from a broken backend. Key changes: - Attach a BackendTrafficPolicy with panicThreshold 0 to routes carrying a networkService backend, and remove it when the backend changes kind - Serve an offline page for responses flagged as having no healthy upstream, ahead of the existing generic mapper, which stays the fallback - Source the offline body the same way as the generic one: optional mount, embedded default, and a missing body falls through to the generic page rather than erroring, so the failOpen:false hook is never stalled
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prototype for Network Services. Draft: this is a working prototype to prove the design, not a merge candidate.
What a consumer can do
Declare a
NetworkServicenaming the members of their application and point an HTTPProxy at it. The edge then serves each request from members in the nearest location and moves on when one fails.Membership tracks whatever is running. A consumer writes no addresses, no per-location weights and no failover order, and adding capacity or a location needs no edit to either resource.
What is here
NetworkService, selecting network interface claims by label, with per-location member and health counts in its status.What the end-to-end run proves
A real request returns 200 through the edge from a member on its allocated address. Envoy receives each member in the locality its claim names, and a member whose data plane is not programmed is carried but marked unhealthy and serves nothing.
What it does not prove yet
The environment has one cell, so ranking across locations cannot be exercised end to end. Multi-location rollup is covered by controller tests only. Proving it for real needs a second cell.
Two known gaps, both recorded in the enhancement: no location carries coordinates yet, so the ranking that makes "nearest" mean anything cannot be computed; and claims are not published to the consumer's project, so membership currently resolves where the claims already are.
Notable limits
A service spans one network. Past 100 members the proxy serves the first shard and reports that it is doing so rather than silently truncating. Health is judged on request outcomes, so a member is ejected after failing rather than before.
Related
feat: allocate a network's fabric identity #412 is now carried on this branch so both can be validated together
feat: derive VPC identifier from the network's fabric identity cloud#11 consumes the network identity this allocates
Read the HolderAvailable contract from the networking API compute#254 reads the holder condition this branch defines, rather than matching it by string
Network Services: load balance an application across the locations it runs in enhancements#870 is the design
feat: take a VPC identifier from the network cloud#16 makes every location of a network share one VPC identifier. Until it lands, a service with members in two locations binds no VRF and fails every request, because a binding is a property of the whole cluster and two identifiers have no correct answer.