proxy: scope datapath rules to the node hosting the backend - #18
proxy: scope datapath rules to the node hosting the backend#18mattia-eleuteri wants to merge 2 commits into
Conversation
Every node programmed the svc_pod/pod_svc maps and the port-filter sets for every service, so a node that did not host the backend still rewrote the destination of a packet leaving it. That premature DNAT breaks conntrack on the owning node. It records the flow as (srcSvcIP -> podIP) because the destination was already translated upstream, while the reply leaves the pod and gets its source rewritten to the service IP by egress_snat, which runs at prerouting priority raw, before conntrack. The tuple (dstSvcIP -> srcSvcIP) matches nothing, the reply is not established, and port_filter drops it since the initiator's ephemeral port is not in its own allowed_ports. The visible effect is that a PortList (wholeIP=false) backend can no longer open a connection to another cozy-proxy managed backend on a different node: the SYN arrives, the SYN-ACK is generated and dropped, and the caller hangs. Traffic from outside the cluster is unaffected, because it is not translated before reaching the owning node. Program the rules only where the backend pod runs, keyed on the endpoint's NodeName against NODE_NAME. Every node then sees a consistent conntrack view, and the maps only carry local entries. Rules for a pod that moved away are withdrawn, both on endpoint events and by the startup cleanup, so state inherited from a cluster-wide build is purged on upgrade. NODE_NAME is read from the environment; when it is absent the check is disabled and the previous cluster-wide behavior is kept, so the binary still runs under a chart that does not inject it yet. Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
📝 WalkthroughWalkthroughChangesNode-aware service rules
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant main
participant ServicesController
participant EndpointEvents
participant Proxy
main->>ServicesController: Set NodeName
EndpointEvents->>ServicesController: Add or update endpoint
ServicesController->>ServicesController: Check endpoint ownership
ServicesController->>Proxy: Apply local rules or withdraw remote rules
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
CleanupRules queued deletions and additions into a single batch and treated any flush error as fatal. Deleting a set element that is already gone reports ENOENT, which fails the whole flush, so the controller returned an error, the manager exited, and the DaemonSet pod entered CrashLoopBackOff with the node's datapath left half-programmed. Scoping the rules to the local node made this reliable rather than rare: the first startup after the change deletes every entry the node inherited for backends it does not host, which is most of them. Commit deletions separately from additions, tolerate ENOENT on the flush the way DeleteRules, DeletePortFilter and DeleteICMPAllow already do, and apply the same split to CleanupPortFilters and CleanupICMPAllow. A cleanup failure is now logged instead of aborting Start, since the informers converge on the next event anyway and staying up with stale entries beats exiting with a partial ruleset. Observed on a 3-node cluster carrying 15 managed services: the transition logs "Ignoring ENOENT on flush" for the cleanup deletions and completes with zero restarts, where it previously crash-looped. Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
|
Follow-up commit: the first rollout of this branch on a production cluster crash-looped one node, which surfaced a second defect worth fixing here rather than separately.
Scoping the rules to the local node turns this from rare into reliable: the first startup after the change deletes every entry the node inherited for backends it does not host, which is most of them. The fix commits deletions separately from additions and tolerates Worth noting that the same Re-validated on a 3-node cluster carrying 15 managed services, transitioning from
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (2)
pkg/controllers/services_controller_test.go (2)
95-116: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd node-scope coverage for startup cleanup.
These tests call
applyRulesdirectly. They do not verify the changedcleanupRemovedServiceskeep sets. Add a test with local and remote endpoints. Assert thatCleanupRules,CleanupPortFilters, andCleanupICMPAllowretain only local pairs. Also verify that an empty controller node name retains all pairs.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/controllers/services_controller_test.go` around lines 95 - 116, Add node-scope coverage for cleanupRemovedServices using service endpoint pairs from both local and remote nodes. Verify CleanupRules, CleanupPortFilters, and CleanupICMPAllow retain only pairs belonging to the controller’s NodeName, and add a separate empty-NodeName case confirming all pairs are retained.
48-50: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the deleted mapping identity.
The test only verifies that
DeleteRuleswas called. It passes if the implementation deletes(svcIP, newPodIP)instead of the stale(svcIP, oldPodIP)mapping. Record thesvcIPandpodIParguments, then assert deletion of192.0.2.10and10.0.0.1.Also applies to: 122-127
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/controllers/services_controller_test.go` around lines 48 - 50, Update recordingProxy.DeleteRules to record the svcIP and podIP arguments in addition to the call marker, then strengthen the relevant test assertions to verify deletion of svcIP 192.0.2.10 with oldPodIP 10.0.0.1 rather than only checking that DeleteRules was called.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/controllers/services_controller.go`:
- Around line 152-160: Update the reconciliation flow around EnsureRules and
withdrawRules so failures from Proxy.EnsureRules and Proxy.DeleteRules are
handled instead of discarded: log the error and enqueue the affected service/IP
pair for retry. Return immediately after a failed EnsureRules so
reconcilePortFilter and successful state recording do not proceed, and apply the
same retry recovery path to failed withdrawals.
- Around line 270-278: Update the startup reconciliation flow around
cleanupRemovedServices so a failed cleanup schedules bounded retry attempts
after informer synchronization, while preserving non-fatal startup behavior.
Reuse the existing controller scheduling, retry, and logging mechanisms if
available, and ensure retries stop after the configured bound or succeed instead
of waiting for a later informer event.
- Around line 106-115: Plan migration from the deprecated v1.Endpoints API to
discoveryv1.EndpointSlice in the service reconciliation flow, aggregating all
slices and endpoints for each service instead of using only the first
subset/address; update endpointNode and its callers accordingly. In
pkg/controllers/services_controller.go#L106-L115, replace the Endpoints-based
lookup with EndpointSlice-aware aggregation. In
pkg/controllers/services_controller_test.go#L15-L22, update fixtures and
coverage to exercise aggregated EndpointSlice data; both sites require changes.
In `@pkg/proxy/nft.go`:
- Around line 606-612: Update cleanupTolerateENOENT and the CleanupPortFilters,
CleanupRules, and CleanupICMPAllow deletion flows so one ENOENT cannot abort
deletion of remaining stale entries. Delete elements individually or re-list and
retry with a reduced batch after ENOENT, ensuring all undeclared entries are
removed. Add a regression test covering multiple stale entries where one is
already absent.
- Around line 530-547: Restrict flushTolerateENOENT to deletion commits, because
suppressing ENOENT during additions can report success when nftables objects are
missing. Update the addition paths around SetAddElements at the three call sites
to use a non-tolerant flush or rebuild the missing objects before retrying,
while preserving tolerant handling for deletions. Add a regression test covering
ENOENT from an addition-only flush.
---
Nitpick comments:
In `@pkg/controllers/services_controller_test.go`:
- Around line 95-116: Add node-scope coverage for cleanupRemovedServices using
service endpoint pairs from both local and remote nodes. Verify CleanupRules,
CleanupPortFilters, and CleanupICMPAllow retain only pairs belonging to the
controller’s NodeName, and add a separate empty-NodeName case confirming all
pairs are retained.
- Around line 48-50: Update recordingProxy.DeleteRules to record the svcIP and
podIP arguments in addition to the call marker, then strengthen the relevant
test assertions to verify deletion of svcIP 192.0.2.10 with oldPodIP 10.0.0.1
rather than only checking that DeleteRules was called.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 43c014a3-f836-40ae-9c30-de690e48e501
📒 Files selected for processing (4)
main.gopkg/controllers/services_controller.gopkg/controllers/services_controller_test.gopkg/proxy/nft.go
| // endpointNode returns the node hosting the endpoint's first address. | ||
| func endpointNode(ep *v1.Endpoints) (string, bool) { | ||
| if !hasValidEndpointIP(ep) { | ||
| return "", false | ||
| } | ||
| node := ep.Subsets[0].Addresses[0].NodeName | ||
| if node == nil || *node == "" { | ||
| return "", false | ||
| } | ||
| return *node, true |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Identify the migration surface before changing endpoint reconciliation.
rg -n -C 2 '\bv1\.(Endpoints|EndpointSubset)\b' \
pkg/controllers/services_controller.go \
pkg/controllers/services_controller_test.go
# Confirm the declared Kubernetes API dependency version.
rg -n 'k8s\.io/api' go.modRepository: cozystack/cozy-proxy
Length of output: 7094
🌐 Web query:
Kubernetes 1.33 Endpoints API deprecated EndpointSlice replacement
💡 Result:
As of Kubernetes 1.33, the Endpoints API is officially deprecated in favor of the EndpointSlice API [1][2][3]. Key details regarding this change: Official Deprecation: The v1 Endpoints API has been marked as deprecated starting in Kubernetes 1.33 [1][4]. The Kubernetes API server now issues warnings to users who read from or write to Endpoints resources [1][5]. Reasoning: The Endpoints API is considered a legacy resource that lacks support for modern service features, such as dual-stack networking, advanced traffic distribution, and scalability for services with large numbers of endpoints [1][4][5]. The EndpointSlice API was introduced to address these limitations and provide a more scalable way to track network endpoints [1][6]. Future of the Endpoints API: While deprecated, the Endpoints API is expected to remain available to maintain backward compatibility for existing workloads and scripts [1][3]. However, it is no longer the recommended practice for managing service traffic [1][2]. Migration: Users and controllers are encouraged to transition to the discovery.k8s.io/v1 EndpointSlice API [1][7]. For most use cases, migrating involves updating YAML manifests or code to use the EndpointSlice resource type, which organizes endpoint information differently than the Endpoints API (e.g., using conditions like ready: false instead of separate lists for ready and not-ready endpoints) [1]. Mirroring: To maintain compatibility, the control plane continues to mirror most user-created Endpoints resources to corresponding EndpointSlices, though this mirroring behavior is also deprecated alongside the Endpoints API itself [6].
Citations:
- 1: https://kubernetes.io/blog/2025/04/24/endpoints-deprecation/
- 2: https://kubernetes.io/docs/reference/kubernetes-api/core/endpoints-v1/
- 3: Mark v1.Endpoints deprecated kubernetes/kubernetes#130098
- 4: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/4974-deprecate-endpoints/README.md
- 5: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/4974-deprecate-endpoints
- 6: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
- 7: https://kubernetes.io/docs/reference/using-api/deprecation-guide/
Plan migration from the deprecated Endpoints API.
Kubernetes deprecated v1.Endpoints in Kubernetes 1.33; discoveryv1.EndpointSlice is the replacement API. This dependency is already at k8s.io/api v0.36.2, so plan EndpointSlice reconciliation when supporting clusters that stop serving Endpoints. EndpointSlice replacement must aggregate slices and endpoints per service rather than reading only the first v1.Endpoints subset/address.
Affected sites:
pkg/controllers/services_controller.go#L106-L115pkg/controllers/services_controller_test.go#L15-L22
🧰 Tools
🪛 golangci-lint (2.12.2)
[error] 107-107: SA1019: v1.Endpoints is deprecated: This API is deprecated in v1.33+. Use discoveryv1.EndpointSlice.
(staticcheck)
📍 Affects 2 files
pkg/controllers/services_controller.go#L106-L115(this comment)pkg/controllers/services_controller_test.go#L15-L22
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/controllers/services_controller.go` around lines 106 - 115, Plan
migration from the deprecated v1.Endpoints API to discoveryv1.EndpointSlice in
the service reconciliation flow, aggregating all slices and endpoints for each
service instead of using only the first subset/address; update endpointNode and
its callers accordingly. In pkg/controllers/services_controller.go#L106-L115,
replace the Endpoints-based lookup with EndpointSlice-aware aggregation. In
pkg/controllers/services_controller_test.go#L15-L22, update fixtures and
coverage to exercise aggregated EndpointSlice data; both sites require changes.
Source: Linters/SAST tools
| c.Proxy.EnsureRules(svcIP, podIP) | ||
| c.reconcilePortFilter(svc, svcIP, podIP, ctx) | ||
| } | ||
|
|
||
| // withdrawRules removes every datapath entry for the pair. Absent entries are | ||
| // not an error. | ||
| func (c *ServicesController) withdrawRules(svcIP, podIP, ctx string) { | ||
| c.clearPortFilter(svcIP, podIP, ctx) | ||
| c.Proxy.DeleteRules(svcIP, podIP) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Handle failed proxy reconciliation.
Lines 152 and 160 discard errors from EnsureRules and DeleteRules. If an nftables operation fails, the controller can retain stale rules or record a service state without its required rules. Do not reconcile port filters after EnsureRules fails. Log the error and enqueue the affected pair for retry. Apply the same recovery path to failed withdrawals.
🧰 Tools
🪛 golangci-lint (2.12.2)
[error] 152-152: Error return value of c.Proxy.EnsureRules is not checked
(errcheck)
[error] 160-160: Error return value of c.Proxy.DeleteRules is not checked
(errcheck)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/controllers/services_controller.go` around lines 152 - 160, Update the
reconciliation flow around EnsureRules and withdrawRules so failures from
Proxy.EnsureRules and Proxy.DeleteRules are handled instead of discarded: log
the error and enqueue the affected service/IP pair for retry. Return immediately
after a failed EnsureRules so reconcilePortFilter and successful state recording
do not proceed, and apply the same retry recovery path to failed withdrawals.
Source: Linters/SAST tools
| // Run cleanup for removed services. A failure here is logged but does not | ||
| // abort: exiting takes the pod down and leaves the node's datapath | ||
| // half-programmed, whereas the informers below converge on the next event. | ||
| log.Info("running cleanup for removed services") | ||
| if err := c.cleanupRemovedServices(); err != nil { | ||
| return fmt.Errorf("failed to cleanup removed services: %w", err) | ||
| log.Error(err, "cleanup of removed services failed, continuing with reconciliation") | ||
| } else { | ||
| log.Info("cleanup of removed services completed") | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Retry failed startup cleanup.
The informer caches are already synced before this call. A cleanup failure is only logged, and cleanupRemovedServices is not called again until a later event or the 12-hour informer resync. Stale remote mappings can remain active after a transient nftables failure. Keep startup non-fatal, but schedule bounded cleanup retries after synchronization.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/controllers/services_controller.go` around lines 270 - 278, Update the
startup reconciliation flow around cleanupRemovedServices so a failed cleanup
schedules bounded retry attempts after informer synchronization, while
preserving non-fatal startup behavior. Reuse the existing controller scheduling,
retry, and logging mechanisms if available, and ensure retries stop after the
configured bound or succeed instead of waiting for a later informer event.
| // flushTolerateENOENT commits the pending batch and treats ENOENT as success. | ||
| // | ||
| // Deleting a set element that is already gone reports ENOENT, which fails the | ||
| // whole flush. Deletions must therefore be committed on their own, so a stale | ||
| // element cannot mask a genuine failure among the additions that would | ||
| // otherwise share the batch. | ||
| func (p *NFTProxyProcessor) flushTolerateENOENT(op string) error { | ||
| err := p.conn.Flush() | ||
| if err == nil { | ||
| return nil | ||
| } | ||
| if errors.Is(err, unix.ENOENT) { | ||
| log.Info("Ignoring ENOENT on flush — element already gone", "op", op) | ||
| return nil | ||
| } | ||
| return err | ||
| } | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Locate file"
git ls-files | rg '(^|/)nft\.go$|(^|/)go\.mod$|(^|/)go\.sum$' || true
echo
echo "## nft.go relevant sections"
if [ -f pkg/proxy/nft.go ]; then
wc -l pkg/proxy/nft.go
sed -n '500,570p' pkg/proxy/nft.go
sed -n '610,660p' pkg/proxy/nft.go
sed -n '830,860p' pkg/proxy/nft.go
sed -n '960,1005p' pkg/proxy/nft.go
else
echo "pkg/proxy/nft.go not found"
fi
echo
echo "## Find flushTolerateENOENT usages"
rg -n "flushTolerateENOENT|SetAddElements|DelElements|Flush\(" pkg/proxy/nft.go pkg/controllers/services_controller.go || true
echo
echo "## Go mod for nftables"
if [ -f go.mod ]; then rg -n "github.com/google/nftables" go.mod go.sum; fi
if [ -f pkg/proxy/go.mod ]; then rg -n "github.com/google/nftables" pkg/proxy/go.mod pkg/proxy/go.sum; fiRepository: cozystack/cozy-proxy
Length of output: 11077
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Determine Conn.Flush ENOENT semantics from vendored/module source if present"
rg -n "func \(c \*Conn\) Flush|ENOENT|ErrTableNotFound|ErrSetNotFound|Flush" $(go env GOPATH 2>/dev/null)/pkg/mod/github.com/google/nftables* 2>/dev/null || true
echo
echo "## Try go doc if module cache has it"
go doc github.com/google/nftables Conn.Flush 2>/dev/null || true
go doc github.com/google/nftables ErrTableNotFound 2>/dev/null || true
go doc github.com/google/nftables ErrSetNotFound 2>/dev/null || trueRepository: cozystack/cozy-proxy
Length of output: 34982
Do not suppress ENOENT for addition commits.
flushTolerateENOENT is used after the SetAddElements calls at Lines 648, 854, and 997. An ENOENT from an addition-only flush can mean the target table or set is missing; returning nil reports successful reconciliation while the desired NAT, port-filter, or ICMP rules are not installed.
Use the tolerant helper only for deletion commits. Use a non-tolerant flush for additions, or rebuild/reinitialize the missing nftables objects before retrying. Add a regression test for an ENOENT addition.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/proxy/nft.go` around lines 530 - 547, Restrict flushTolerateENOENT to
deletion commits, because suppressing ENOENT during additions can report success
when nftables objects are missing. Update the addition paths around
SetAddElements at the three call sites to use a non-tolerant flush or rebuild
the missing objects before retrying, while preserving tolerant handling for
deletions. Add a regression test covering ENOENT from an addition-only flush.
| // Commit the deletions before queueing the additions below: an element | ||
| // that is already gone fails the flush, and a shared batch would report | ||
| // that as a cleanup failure, which aborts the controller at startup. | ||
| if err := p.flushTolerateENOENT("CleanupRules deletions"); err != nil { | ||
| log.Error(err, "Failed to commit cleanup deletions") | ||
| return fmt.Errorf("failed to commit cleanup deletions: %v", err) | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate nft.go and relevant symbols"
git ls-files | rg '(@|^)pkg/proxy/nft\.go$|pkg/proxy/interface\.go$' || true
echo
echo "Outline nft.go around relevant symbols"
ast-grep outline pkg/proxy/nft.go --view expanded | sed -n '1,220p' | rg -n "Cleanup|flush|nftables|Remove|Add|Batch|ENOENT|Set|Entry" -C 2 || true
echo
echo "Show relevant lines"
sed -n '500,630p' pkg/proxy/nft.go
sed -n '800,860p' pkg/proxy/nft.go
sed -n '950,1005p' pkg/proxy/nft.go
echo
echo "Search for flushTolerateENOENT definitions/usages"
rg -n "flushTolerateENOENT|ENOENT|torture|errno|ENOENT" pkg/proxy/nft.go pkg/proxy/interface.goRepository: cozystack/cozy-proxy
Length of output: 12946
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Show full cleanup functions relevant to deletion batching"
sed -n '625,670p' pkg/proxy/nft.go
sed -n '750,860p' pkg/proxy/nft.go
sed -n '946,1002p' pkg/proxy/nft.go
echo
echo "Show current list handling before deletes in CleanupPortFilters"
sed -n '752,832p' pkg/proxy/nft.go
echo
echo "Search for regression tests around stale ENOENT cleanup"
rg -n "Cleanup.*(ENOENT|stale|already gone)|ENOENT.*Cleanup|stale.*(PortFilter|allowed|ICMP|Cleanup|CleanupRules)" -g '*_test.go' pkg/proxy || true
echo
echo "Read-only verifier: model ENOENT on one stale element in mixed stale/good deletion array"
python3 - <<'PY'
elements = ["stale-a", "good", "stale-b"]
def delete_batch_delete_many(batch):
stale = [e for e in batch if e.startswith("stale")]
if stale:
# nftables batch aborts when one element is already absent;
# we model the return as ENOENT without knowing which element failed.
raise ENOENT("element already absent")
return []
class ENOENT(Exception):
pass
try:
delete_batch_delete_many(elements)
except ENOENT:
remaining = [e for e in elements if e.startswith("stale")]
print("ENOENT on stale/good batch deletes remaining stale=", remaining, "retains stale=", remaining)
PYRepository: cozystack/cozy-proxy
Length of output: 11009
Retry or isolate deletion batches after ENOENT.
cleanupTolerateENOENT accepts ENOENT as success for CleanupPortFilters, so SetDeleteElements(p.allowedPorts, delPorts) can fail anywhere in the batch and leave other stale allowedPorts keys behind. If an already-absent element aborts a batch that also deletes undeclared allowed ports, no retry or re-list happens, so undeclared ports can remain accepted. CleanupRules and CleanupICMPAllow batch multiple elements the same way. Delete stale entries per element, or re-list after ENOENT and build a smaller deletion batch; add a regression test with multiple stale entries where one is already absent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/proxy/nft.go` around lines 606 - 612, Update cleanupTolerateENOENT and
the CleanupPortFilters, CleanupRules, and CleanupICMPAllow deletion flows so one
ENOENT cannot abort deletion of remaining stale entries. Delete elements
individually or re-list and retry with a reduced batch after ENOENT, ensuring
all undeclared entries are removed. Add a regression test covering multiple
stale entries where one is already absent.
1 participant
Problem
Every node programs the
svc_pod/pod_svcmaps and the port-filter sets for every service, not just the ones whose backend it hosts. A node that does not host the backend therefore rewrites the destination of a packet that is merely leaving it.That premature DNAT desynchronises conntrack on the owning node:
egress_snatruns at prerouting priorityraw(-300), before conntrack (-200)ingress_dnatruns at prioritymangle(-150), after conntrackport_filterruns at priorityfilter(0) and relies onct state established,related acceptto let replies throughFor a flow coming from outside the cluster this is consistent: conntrack records
(client -> svcIP), the reply is SNATed back tosvcIPbefore conntrack sees it, the tuple matches,port_filteraccepts.For a flow initiated by another cozy-proxy managed backend it is not. The source node already translated the destination, so the owning node records
(srcSvcIP -> podIP). The reply leaves the pod andegress_snatrewrites its source todstSvcIPbefore conntrack, producing(dstSvcIP -> srcSvcIP), which matches nothing. The reply is notestablished, falls through to the drop rule, and the initiator's ephemeral port is of course not in its ownallowed_ports. Dropped.Visible effect: a PortList (
wholeIP: "false") backend can no longer open a connection to another managed backend on a different node. The SYN arrives, the SYN-ACK is generated and silently dropped, the caller hangs. Same node works. Egress to the internet works. Ingress from outside works. Only the cross-node managed-to-managed path is broken.Reproduction
A plain pod behind a LoadBalancer Service carrying the cozy-proxy label and
wholeIP: "false"is treated exactly like a VM, which makes this reproducible without KubeVirt:http=200wholeIP: "true"(removing it fromfiltered_pods): works again immediatelySYN_SENT ... [UNREPLIED], while a control flow to the internet is[ASSURED]Fix
Program the rules only on the node hosting the backend, keyed on the endpoint's
NodeNameagainstNODE_NAME. Every node then has a consistent conntrack view, and the maps only carry local entries instead of a full copy of the cluster's services.Rules for a pod that moved away are withdrawn, both on endpoint events and by the startup cleanup, so state inherited from a cluster-wide build is purged on upgrade rather than lingering.
NODE_NAMEis read from the environment. When it is absent the check is disabled and the previous cluster-wide behavior is kept, so this binary still runs under a chart that does not inject the variable yet. The chart needs a matching change to set it fromspec.nodeName; without it the fix is inert (but nothing breaks).Validation
Controlled A/B/A on a 3-node cluster, cross-node initiator and target:
Reverting the image to v0.3.0 reproduces the hang, re-applying the fix clears it. Port filtering from outside is unchanged, so the security property the port filter exists for is preserved.
Per-node mapping count drops from "every service in the cluster" to "the backends on this node".
Unit tests added for
servesEndpoint, the withdraw-on-remote-backend path, and stale endpoint withdrawal on pod IP change.Summary by CodeRabbit
New Features
Bug Fixes