ci: add an IPv6-only kind e2e job [DO NOT MERGE — signal only] - #939
Draft
Yuan Gao (ygao-g) wants to merge 6 commits into
Draft
ci: add an IPv6-only kind e2e job [DO NOT MERGE — signal only]#939Yuan Gao (ygao-g) wants to merge 6 commits into
Yuan Gao (ygao-g) wants to merge 6 commits into
Conversation
Yuan Gao (ygao-g)
force-pushed
the
ci-kind-ipv6
branch
4 times, most recently
from
August 14, 2026 18:04
2d6ebff to
72285e4
Compare
IP_FAMILY selects ipv4, ipv6 or dual and becomes networking.ipFamily, leaving kind's per-family subnet defaults alone. The script also recreates a pre-IPv6 "kind" Docker network, fails fast if the daemon has IPv6 off, sets proxy_ndp alongside proxy_arp for gVisor pod-to-pod traffic, and repoints an ipv6 kubeconfig from [::1] at localhost so a client outside the Docker host can still reach the apiserver. Tested on kind with all three families: node InternalIPs, Service ClusterIPs and pod IPs land in the requested families, pod-to-pod and CoreDNS work on them, and pods still pull through the local registry.
CoreDNS runs dnsPolicy: Default and inherits the node's Docker-generated /etc/resolv.conf, which always names an IPv4 resolver -- unreachable from a v6-only pod, so every external lookup dies at "connect: network is unreachable". Behind that sits a second failure: this script wires the registry into containerd on the node, but atelet pulls actor images from its own pod netns, where kind-registry does not resolve at all. Rewrite the Corefile on IPv6-only clusters: a hosts block mapping kind-registry to the registry's GlobalIPv6Address, with fallthrough so non-registry names still reach the new IPv6 forwarder. Hard-fail if the substitution was a no-op, and probe both names from a pod rather than the node, which is dual-stack and would resolve them either way.
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and nothing at all on its IPv6 one. Pair each primary socket with an additional "::" address on the same port. Ipv4Compat is false on the additional address. Setting it would clear IPV6_V6ONLY and collide with the primary IPv4 wildcard already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind -- that would take down all ingress, not just the IPv6 half. Hoisting the literal into a helper keeps the two listeners from drifting. No behaviour change on an IPv4-only cluster: the primary address is untouched, and a host without IPv6 simply has no second socket to bind.
The Envoy admin socket bound 0.0.0.0, leaving it reachable over IPv4 only. It binds "::" with ipv4_compat now -- one socket for both families. ipv4_compat is required rather than incidental here: Envoy sets IPV6_V6ONLY without it, and dataplane.go health-checks the listener over http://127.0.0.1:9901/ready, so dropping it would take the dataplane component of /statusz unhealthy. The atenet-router Service carried no ipFamilyPolicy, which the API server defaults to SingleStack -- an IPv4 ClusterIP and nothing else, which leaves the listeners above with no IPv6 address to answer on. Prefer, not Require, so this stays valid on a single-stack cluster, where it is a no-op. spec.ipFamilies is deliberately left alone: the primary family is immutable and the API server appends the secondary one itself.
…families The gateway's two Envoy sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet's startup probe against the admin port was refused and atenet-egress crashlooped while Envoy itself started fine and logged "admin address: 0.0.0.0:15000". The :443 listener had the same gap, leaving no v6 path for an actor's CONNECT. Both are single sockets, so they bind "::" with ipv4_compat rather than taking the additional-address pairing the ingress listeners use. On the admin socket ipv4_compat is load-bearing: the ext-proc sidecar's drainer reaches it at 127.0.0.1:15000 and envoydrain.go reads a refusal as "Envoy already exited", so a bare "::" would silently skip the drain. The Service gets PreferDualStack for the same reason the router's does -- without it a dual-stack cluster hands out one ClusterIP and the new v6 bind is unreachable.
Runs the full install plus the demo and networking e2e suites against a single-stack IPv6-only kind cluster, and asserts the cluster really is v6-only so a green run cannot quietly become a second IPv4 run. It stays out of the e2e-test merge gate, so it reports IPv6 status without being able to block a PR, and it runs on every PR for now so the results are visible; the TODO on the trigger records the intended ci/ipv6 label gate. ubuntu-latest has no IPv6 egress, so the job stands up tayga for NAT64 and points CoreDNS at an upstream resolver through the well-known prefix. DNS64 is scoped to a catch-all server block: synthesizing AAAA over the cluster zones destroys the v6-only ClusterIP answers and the control plane never comes up.
Yuan Gao (ygao-g)
force-pushed
the
ci-kind-ipv6
branch
from
August 14, 2026 18:13
72285e4 to
68972e5
Compare
Yuan Gao (ygao-g)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not for merging in its current state. This draft exists to produce IPv6-only e2e signal on demand — for current HEAD plus the pending IPv6 stack — because no such coverage exists anywhere in CI today. It carries its dependencies in the diff and runs on every PR; neither is a shape that should land. The run results are the deliverable here, not the branch.
The job builds a single-stack v6 kind cluster with NAT64 + DNS64 on the runner, does a full
--deploy-ate-systeminstall, and runs thedemoandnetworkingsuites. It asserts the cluster really is single-stack before testing anything, and guards against a vacuously green run.It is red, and that is the point. At this HEAD
TestActorEgressfails: an actor has no IPv6 stack, because the sandbox netns gets a hardcoded IPv4 link-local /30 with v4-only masquerade (internal/ateomnet/net.go). Issue to follow. The other failure,TestDurableDirLifecycle, is a pre-existing cert-minting flake — the samedemo_test.go:543PermissionDeniedfires on the IPv4 job. Results:demo17 pass / 5 skip,networking3 pass / 1 fail. Coverage is deliberately narrower than the IPv4 job — 2 of 7 suites, gVisor only,certauth only, single node, no CSI.Stacked on #877 (the
IP_FAMILYknob), #958 (DNS on v6-only clusters), and #911 (all three Envoy/Service dual-stack commits) — GitHub will not take a base branch that lives only in a fork, so those ride along and drop out as each merges. Once they have all landed this reduces to a single workflow file with a real trigger — aci/ipv6label gate (see theTODO(246)in the header) or apaths:filter — and that is the version worth reviewing for merge. Could a maintainer create theci/ipv6label in the meantime? The job stays out of thee2e-testgate regardless, so it never blocks a PR. Part of #246.