feat: implement VPC ingress control-plane backend (#856) - #387
Merged
Conversation
Implements steps 1-2 of the VPC ingress control-plane plan (#856): a new backend kind that references a CNI-published EndpointSlice as a Gateway API backend instead of synthesizing one, so the pod address stays joined to the SID annotation the tenant-VRF/SRv6 mechanism on the data plane depends on. Key changes: - Add HTTPProxyRuleBackend.VPCPod, mutually exclusive with endpoint and connector via a new CEL oneOf; relax endpoint to optional - Bypass the unconditional endpoint URL-parsing and EndpointSlice synthesis in collectDesiredResources for vpcPod backends; build the backendRef directly from the referenced EndpointSlice's name/port - Surface a missing referenced EndpointSlice as a Programmed=False/ VPCPodBackendNotFound condition with a short requeue, instead of a bare generic requeue - Extend httpproxy_validation.go with vpcPod field validation, splitting the endpoint-specific checks into their own function since vpcPod backends don't carry an endpoint URL at all TODO(#856): the tenant-id label name/schema this backend implicitly depends on downstream, and whether the referenced EndpointSlice lives in the HTTPProxy's own namespace via the same client used here, are both unconfirmed with #854. Revisit once that's settled.
Implements step 3 of the VPC ingress control-plane plan (#856): a tenant-labeled EndpointSlice backendRef skips the usual downstream Service + EndpointSlice synthesis and BackendTLSPolicy handling, and resolves the CNI-published EndpointSlice directly off the downstream cluster instead. Envoy needs a real pod address for the tenant-VRF/ SRv6 socket-bind mechanism to work, so a ClusterIP Service in the middle defeats the whole point. Key changes: - Branch on a new VPCPodTenantIDLabel before the existing unconditional upstream Get + GC-finalizer stamp, not just before Service synthesis further down — a CNI-published EndpointSlice's lifecycle belongs to galactic-cni, not this controller, so it gets neither - Add passThroughVPCPodBackendRef to resolve the downstream-native EndpointSlice and pass the backendRef through unmodified - Add a placeholder downstream watch (listGatewaysForDownstreamVPCPodEndpointSlice) so a CNI-side pod add/remove re-triggers the owning Gateway; flagged inline as the one piece of this integration with no existing precedent in this codebase TODO(#856): both the tenant-id label and the assumption that galactic-cni publishes into the same mapped downstream namespace this controller already computes are unconfirmed with #854. The watch implementation is explicitly a placeholder pending isolated prototyping, not a designed solution.
Implements step 4 of the VPC ingress control-plane plan (#856): the extension server now watches the CNI-published EndpointSlice a vpcPod HTTPProxy backend references and patches the resulting Envoy cluster with a VRF socket-bind option, so the shared multi-tenant fleet routes each backend through the right tenant's VRF/SRv6 device. Key changes: - Prime EndpointSlice into the edge cache and add a manual RBAC grant for it (config/extension-server/rbac/role.yaml is hand-maintained; make manifests does not cover it) - Index vpcPod backends the same way Connector backends already are (upstreamNS, httpProxyName, ruleIndex) -> tenant ID, resolved from the referenced EndpointSlice's tenant-id label - Add mutate.ApplyVPCPodSocketBind: patches upstream_bind_config with a generic SOL_SOCKET/SO_BINDTODEVICE socket option at STATE_PREBIND, matching clusters via the same httproute/<dsNS>/<proxy>/rule/<idx> name pattern connector clusters already use - Bound the VRF device name to IFNAMSIZ-1 (15 bytes); SO_BINDTODEVICE silently fails to bind past that Caught during testing: protojson.Unmarshal resets its destination message before decoding rather than merging into it, so the bind config is built into its own fresh message and assigned onto the cluster's UpstreamBindConfig field directly, instead of unmarshaling the patch straight into the existing cluster (which would have silently wiped every field Envoy Gateway had already set). TODO(#856): the VRF device-naming scheme is a placeholder pending #855's actual convention. The tenant-id label and the assumption that the referenced EndpointSlice lives in the same local namespace as the HTTPProxy replica are both unconfirmed with #854.
Implements step 5 of the VPC ingress control-plane plan (#856), and corrects a mechanism error the plan itself carried: EnvoyDeployment has no field for adding a long-running sidecar container — 'container' customizes only the main Envoy container, and 'initContainers' is run-to-completion, not long-running. The sidecar has to go in via the existing StrategicMerge patch on spec.template.spec.containers instead, which Kubernetes appends to by container name rather than replacing. Key changes: - Add the #855 VRF/SRv6 sidecar via envoyDeployment.patch in both config/dev/downstream_resources/downstream-gateway.yaml and config/e2e-downstream/envoyproxy.yaml (the only two EnvoyProxy CR instances in this repo), with CAP_NET_ADMIN and no other privilege - Add CAP_NET_RAW to the main Envoy container's securityContext — required for its own SO_BINDTODEVICE call (step 4), separate from and easy to miss alongside the sidecar's CAP_NET_ADMIN - Point the raw-patch-fragility comment at Taskfile.test-infra.yml's ENVOY_GATEWAY_VERSION, the actually-deployed EG controller, not go.mod's unrelated Go API module version - Use busybox/sleep as the sidecar image rather than a nonexistent #855 image reference: an unpullable image on a container sharing the Envoy pod would block the whole pod's readiness (ImagePullBackOff gates every container by default), breaking task test-infra:up and make run today. Swap for #855's real binary once it's published. Verified via # Kustomize Validation Results 🔍 Validating: ./config/certmanager ✅ ./config/certmanager is valid 🔍 Validating: ./config/crd/gateway ✅ ./config/crd/gateway is valid 🔍 Validating: ./config/crd/downstream ❌ Error in './config/crd/downstream': ---------------------------------------- Error: accumulating resources: accumulation err='accumulating resources from '../bases/networking.datumapis.com_trafficprotectionpolicies.yaml': security; file '/home/sprygada/datum-cloud/network-services-operator/config/crd/bases/networking.datumapis.com_trafficprotectionpolicies.yaml' is not in or below '/home/sprygada/datum-cloud/network-services-operator/config/crd/downstream'': must build at directory: '/home/sprygada/datum-cloud/network-services-operator/config/crd/bases/networking.datumapis.com_trafficprotectionpolicies.yaml': file is not directory ---------------------------------------- 🔍 Validating: ./config/crd ✅ ./config/crd is valid 🔍 Validating: ./config/default ✅ ./config/default is valid 🔍 Validating: ./config/dev/downstream_resources ✅ ./config/dev/downstream_resources is valid 🔍 Validating: ./config/dev ✅ ./config/dev is valid 🔍 Validating: ./config/e2e ✅ ./config/e2e is valid 🔍 Validating: ./config/iam ✅ ./config/iam is valid 🔍 Validating: ./config/iam/protected-resources ✅ ./config/iam/protected-resources is valid 🔍 Validating: ./config/iam/roles ✅ ./config/iam/roles is valid 🔍 Validating: ./config/manager ✅ ./config/manager is valid 🔍 Validating: ./config/milo/activity ✅ ./config/milo/activity is valid 🔍 Validating: ./config/milo/activity/policies ✅ ./config/milo/activity/policies is valid 🔍 Validating: ./config/milo ✅ ./config/milo is valid 🔍 Validating: ./config/network-policy ✅ ./config/network-policy is valid 🔍 Validating: ./config/prometheus ✅ ./config/prometheus is valid 🔍 Validating: ./config/quota/claim-policies ✅ ./config/quota/claim-policies is valid 🔍 Validating: ./config/quota ✅ ./config/quota is valid 🔍 Validating: ./config/quota/registrations ✅ ./config/quota/registrations is valid 🔍 Validating: ./config/rbac_deployment ✅ ./config/rbac_deployment is valid 🔍 Validating: ./config/rbac_downstream ✅ ./config/rbac_downstream is valid 🔍 Validating: ./config/resource-metrics-policies ✅ ./config/resource-metrics-policies is valid 🔍 Validating: ./config/resource-metrics ✅ ./config/resource-metrics is valid 🔍 Validating: ./config/telemetry/alerts ✅ ./config/telemetry/alerts is valid 🔍 Validating: ./config/tools/cert-manager ✅ ./config/tools/cert-manager is valid 🔍 Validating: ./config/tools/envoy-gateway ✅ ./config/tools/envoy-gateway is valid 🔍 Validating: ./config/tools/envoy-gateway/overlays/e2e ✅ ./config/tools/envoy-gateway/overlays/e2e is valid 🔍 Validating: ./config/tools/external-dns ✅ ./config/tools/external-dns is valid 🔍 Validating: ./config/tools/billing-usage-collector ✅ ./config/tools/billing-usage-collector is valid 🔍 Validating: ./config/tools/envoy-gateway-downstream ✅ ./config/tools/envoy-gateway-downstream is valid 🔍 Validating: ./config/upstream_resources ✅ ./config/upstream_resources is valid 🔍 Validating: ./config/webhook ✅ ./config/webhook is valid 🔍 Validating: ./config/components/service-catalog ✅ ./config/components/service-catalog is valid 🔍 Validating: ./config/components/cell-controllers ✅ ./config/components/cell-controllers is valid 🔍 Validating: ./config/e2e-downstream/eg-downstream ✅ ./config/e2e-downstream/eg-downstream is valid 🔍 Validating: ./config/e2e-downstream/extserver-base ✅ ./config/e2e-downstream/extserver-base is valid 🔍 Validating: ./config/e2e-downstream ✅ ./config/e2e-downstream is valid 🔍 Validating: ./config/extension-server-e2e ✅ ./config/extension-server-e2e is valid 🔍 Validating: ./config/extension-server/certmanager ✅ ./config/extension-server/certmanager is valid 🔍 Validating: ./config/extension-server ✅ ./config/extension-server is valid 🔍 Validating: ./config/extension-server/network-policy ✅ ./config/extension-server/network-policy is valid 🔍 Validating: ./config/extension-server/rbac ✅ ./config/extension-server/rbac is valid 🔍 Validating: ./config/cell ✅ ./config/cell is valid 🔍 Validating: ./config/dependencies/ipam/overlay ✅ ./config/dependencies/ipam/overlay is valid 🔍 Validating: ./config/dependencies/milo/overlay ✅ ./config/dependencies/milo/overlay is valid 🔍 Validating: ./config/dependencies/postgres-operator ✅ ./config/dependencies/postgres-operator is valid 🚨 One or more kustomizations failed validation. and a rendered of both manifests. TODO(#856): sidecar image/binary is a placeholder pending #855.
The vpcPod not-found branch added in the earlier commit pushed Reconcile's cyclomatic complexity to 61, past this repo's gocyclo threshold of 60. Collapse both decisions (err != nil, errors.As) into a single helper call so Reconcile keeps exactly one branch here, same as before vpcPod existed. Also drops a redundant int32->PortNumber conversion (PortNumber is a plain type alias for int32) golangci-lint's unconvert flagged. make lint is clean; make test still passes.
CI caught this: the vpcPod CEL rule required exactly one of endpoint/connector/vpcPod, which rejected every existing connector backend — connector backends have always required endpoint alongside connector (endpoint is the tunnel's target address; connector says which tunnel to route it through). Only vpcPod is actually mutually exclusive with the other two. Failing scenario: test/e2e-edge/connector-offline-503, which sets both endpoint and connector on the same backend, exactly like every other connector-backed HTTPProxy in this repo. New rule: endpoint is required unless vpcPod is set; vpcPod excludes both endpoint and connector. Also adds test/crd/httpproxy_backend_test.go, an envtest-backed regression test that exercises the CEL rule against a real apiserver for all five backend shapes. This is the coverage gap that let the bug ship: internal/validation's unit tests exercise the hand-rolled Go validation, never the CRD's CEL rules, and the fake client used elsewhere never enforces schema at all. Verified the new test fails against the reverted (buggy) rule and passes against the fix.
scotwells
previously approved these changes
Aug 20, 2026
scotwells
left a comment
scotwells
left a comment
Contributor
There was a problem hiding this comment.
This approach is fine for now to verify the integration. I'd like to look at something that uses the EndpointSlice mechanism or similar so we can easily program envoy to spread load across multiple backends.
scotwells flagged the vpcPod field/type naming on HTTPProxyRuleBackend as review feedback: "instance" reads better in the consumer-facing API than an implementation detail like "VPC pod". Renames the public API surface only: the field, its json tag, the InstanceBackendRef type, the CEL exclusivity rule, and the Programmed=False condition reason (visible via kubectl describe). Internal extension-server identifiers (VPCPodTenantIDLabel, ApplyVPCPodSocketBind, the PolicyIndex cache) keep their existing names since they describe the CNI/VRF mechanism, not the schema. Key changes: - HTTPProxyRuleBackend.VPCPod -> Instance (json: vpcPod -> instance) - VPCPodBackendRef type -> InstanceBackendRef - HTTPProxyReasonVPCPodBackendNotFound -> ...InstanceBackendNotFound - errVPCPodBackendNotFound -> errInstanceBackendNotFound - update controllers, validation, and tests for the new field name - regenerate CRDs, deepcopy, and API docs
scotwells
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the plan in #342 for NSO's slice of VPC ingress (enhancements#856, accepted design enhancements#851): let a tenant's HTTPProxy reference the EndpointSlice galactic-cni publishes for a VPC pod, skip synthesizing a downstream Service for it so Envoy gets a real pod address, and bind the resulting Envoy cluster to the tenant's VRF device via
SO_BINDTODEVICE— the mechanism the shared multi-tenant Envoy fleet needs to disambiguate address space that collides across tenants.What changed
HTTPProxyRuleBackend.VPCPodfield, mutually exclusive withendpoint/connector.vpcPodbackend references the existing EndpointSlice instead of synthesizing one; a missing reference surfaces asProgrammed=False/VPCPodBackendNotFoundwith a short requeue instead of a bare generic one.upstream_bind_configwith aSO_BINDTODEVICEsocket option naming the tenant's VRF device.CAP_NET_RAWon the Envoy container for its own bind call.Known placeholders (all flagged inline as
TODO(#856))IFNAMSIZ-1(15 bytes).busybox:1.36/sleep infinitystand-in so the manifests stay deployable (task test-infra:up,make run) before #855 publishes a real one.EndpointSliceRBAC.listGatewaysForDownstreamVPCPodEndpointSlice) has no precedent elsewhere in this codebase and is a coarse first pass (re-reconciles every Gateway in the namespace), not a designed solution.Test plan
make test(envtest, full suite —internal/controller64.3% coverage,extensionserver/cache83.7%,extensionserver/mutate86.6%)make linttask validate-kustomizations+ a renderedkustomize buildof both touched EnvoyProxy manifestsRelated to #342, #856