feat: allocate a network's fabric identity - #12
Merged
Conversation
A network's identity on the fabric is drawn per location, at random, and checked only against the VPCs in the cell that drew it. Two locations of one network therefore get unrelated values by construction. The Route Target is derived from that value, so neither location imports the other's routes, and the VRF device is named from it, so a service spanning locations needs two devices where one socket binds to one. Two networks that collide do not fail closed: they merge, and tenant traffic crosses between them. A network is one network, so the value that says so cannot be decided in any one location. This adds a central component that allocates it once per network and carries it to the cells where the network is required. Key changes: - Add NetworkFabricIdentity, one per Network, spec-only, and a central fabric-identity-controller that allocates and publishes it - Draw the identity from an IPAM block in a platform-owned tenancy and read it from the block's index, so uniqueness, exhaustion accounting and audit come from an allocator that already solves them - Drive everything from NetworkContexts, which name the network, carry its UID and give the locations it is required in - Mark each identity with the locations that need it and keep one policy per location, so the policy count follows locations, not networks - Collect the object when nothing requires it, while retaining the allocation forever so an identity is never reissued - Take an IPAM dependency, porting the per-tenancy client seam rather than inventing a second pattern The identity is 32 bits because that is what survives into the Route Target, and is never zero and never moved.
Naming the identifier claim from the network's UID meant reaching for a UID that nothing else needed, which forced a field onto NetworkContext purely to carry it. Keying on namespace and name removes that without giving up what retention is for. An identifier is still never released, so it can never return to the pool where the allocator could hand it to an unrelated network. A claim that finds its own retained allocation adopts it rather than failing. The cost is that a network deleted and recreated under the same name in the same namespace inherits the identifier it had before. That is confined to one name in one namespace, which is a far narrower risk than reissue to anything at all. Key changes: - Derive the claim name from namespace and name, delimited by a dot, since a namespace cannot contain one and a dash would be ambiguous - Adopt a retained allocation on conflict, reading the block back from the allocation the refusal names - Drop networkRef.uid, which would document the opposite of the rule
The fabric identity controller was driven entirely by NetworkContexts, which was a reachability compromise: this component has no multi-cluster runtime, so it could not watch a Network where one lives. It could therefore never tell a network that had been deleted from one that happens to be required nowhere right now, and it collected the identity in both cases. A live network whose last context drained lost the value the fabric knows it by, and got it back only because the allocation is retained. The Network is now mirrored onto the federation hub alongside its contexts, so the identity's own lifecycle follows the network and only the placement follows the contexts. Key changes: - Key allocation on the Network existing rather than on a context appearing, and collection on the Network being gone - Keep the identity of a live network required nowhere, withdrawing only its placement, so the next location it reaches is served at once - Treat a Network on its way out as still present: the VRF device is named from the identity and the Route Target derives from it, so a teardown in flight still needs it - Read the Network and its contexts from the hub as copies, never their UID, since a copy is a different object from its source - Add a fabric-identity overlay and component so the central controller is actually deployed, with the hub and IPAM credentials mounted non-optional - Install the NetworkFabricIdentity CRD, which was generated but never added to the CRD kustomization
The controller ran its manager against the federation hub, on the grounds that everything it reads and writes is there. That put a constant lease write on a control plane everything else federates through, and tied this component's leadership to that plane being reachable: a hub hiccup would churn leadership and restart the controller. An unreachable hub has to cost this component its work. It does not have to cost it its identity. Key changes: - Run the manager against the cluster the component is scheduled on and reach the hub as a second cluster, the way the cell manager does - Source both watches from the hub's cache, since that is where the mirrored networks and contexts live - Refuse to start without a hub to watch, rather than come up watching a plane holding none of the objects the controller exists for - Declare the lease and event access the local ServiceAccount now needs, which its binding already covers
A cluster carries a cache, so the manager runs it in the cache group rather than the leader election group. It starts on every replica ahead of any election and startup blocks until its cache has synced, which is already the property a probe would assert.
scotwells
marked this pull request as ready for review
August 27, 2026 13:03
scotwells
requested review from
privateip and
savme
and removed request for
savme
August 27, 2026 13:03
privateip
approved these changes
Aug 27, 2026
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A network placed in two locations is two networks to the fabric. Each location draws its own identifier at random and checks it only against its own cell, so the two never match and neither imports the other routes. This allocates the identity once per network and carries it to the cells where the network is required.
The identity is drawn from an address pool that is never routed, so uniqueness, exhaustion accounting and audit come from an allocator that already solves them. It is never reissued: when a network is gone the object is collected and the allocation is retained.
Reading networks needs datum-cloud/network-services-operator#411, which mirrors them onto the federation hub beside the contexts already there.
Enumerability
This makes the identifier guessable where today it is random, because the address service offers no randomised allocation strategy. That matters because the component consuming it authorises nothing. Its authorizer is a permissive stub, and its own code notes that a tenant able to name another tenant VPC could redirect or intercept their traffic.
It is also thinner than it looks. The fabric truncates the identifier to 32 bits on the way into the Route Target, so today wider random value already has only 32 bits of effective entropy. A number that narrow cannot be a secret. The fix is to authorise the identifier rather than hide it, and that is not in this change.
Deployment
The central controller ships with manifests for the first time, following the split this platform already uses for a component that runs centrally rather than in a cell.
It reaches the hub as a second cluster and only the leader election lease is local. An unreachable hub has to cost this component its work. It does not have to cost it its identity.
Related: datum-cloud/enhancements#870