Clarify denyOut semantics for sandbox-local MMDS (169.254.169.254)

[cristianducu]

Question about the security semantics of outbound network controls and sandbox-local MMDS.

In one hosted sandbox diagnostic, created with secure:true, allowPublicTraffic:false and network.denyOut:["169.254.169.254/32"], TCP port80 and the MMDS endpoint remained reachable. Following envd's documented source protocol (PUT /latest/api/token with X-metadata-token-ttl-seconds:60, then GET / with X-metadata-token) returned HTTP200. The JSON keys were exactly instanceID, envID, address and accessTokenHash. The instance/template identifiers matched the sandbox/template used for this diagnostic. No values or credentials are included here. The sandbox was killed afterward.

Pinned source inspected: a1b5bd7, packages/envd/internal/host/mmds.go and mmds_route_linux.go. PinMMDSRoute appears to preserve access to this endpoint for /init verification.

Could you clarify:

1. Is sandbox-local MMDS intentionally exempt from denyOut, including TCP? Is there a supported way to deny user-workload access without breaking envd initialization?
2. Is this address guaranteed to resolve only to sandbox-scoped MMDS, never host/cloud-instance metadata or IAM/service-account credentials, across supported runtime/network configurations?
3. What is the supported, testable isolation boundary for cloud-host metadata credentials? Are there documented protected paths/redirect handling or provider conformance tests we can reference?

This is a request for the supported contract, not a claim of exposed host credentials. Our single diagnostic does not prove the behavior of all paths or deployments.

[AdaAibaby]

Thanks for the careful, well-scoped report — inspection confirms the behavior you saw, and it's by design. Answering each question against the pinned source:

1. Is sandbox-local MMDS intentionally exempt from denyOut, including TCP? Is there a supported way to deny user-workload access without breaking envd init?
   Effectively yes, but not via an allow-rule that overrides deny — MMDS simply isn't on the filtered path. 169.254.169.254 is served by Firecracker's per-microVM MMDS (V2), configured per-sandbox by the orchestrator (fc/client.go PutMmdsConfig/setMmds). Those packets are handled by the VMM's virtio-net device and never egress the tap. Every denyOut/allowPublicTraffic control operates only on tap→host-netns traffic: the nftables filter chain applies user allow/deny sets to non-TCP only (TCP is REDIRECTed to a host-netns proxy), and the TCP proxy enforces denyOut plus a baseline DeniedSandboxCIDRs that already contains 169.254.0.0/16. Since MMDS never crosses the tap, none of these see it — so denyOut:["169.254.169.254/32"] and allowPublicTraffic:false have no effect on it. This is invariant across normal, BYOP, and full deny-egress modes. envd additionally pins a RETURN rule for 169.254.169.254:80 at the top of the guest's nat PREROUTING/OUTPUT (mmds_route_linux.go, self-healing in init.go) to keep /init verification working.
   There is currently no supported knob to deny user-workload access while preserving envd init: MMDS is reachable by any process in the guest, and envd needs it for /init token-hash verification and metadata polling. This is the one real gap the issue surfaces (see below).

2. Is the address guaranteed to resolve only to sandbox-scoped MMDS, never host/cloud metadata or IAM credentials?

   Within the guest, yes: 169.254.169.254 is answered only by Firecracker's per-VM MMDS, whose contents are exactly the four sandbox-scoped keys the orchestrator writes (instanceID, envID, address, accessTokenHash) — it does not proxy host or cloud IMDS. The host's real IMDS lives in the host network namespace; the guest's identically-numbered address is intercepted by the VMM and cannot reach it, and even tap-egress to 169.254.0.0/16 is denied by the proxy and the nftables denylist. One caveat worth stating explicitly: "never routes to a real host/cloud IMDS" is partly a deployment contract — it also depends on the guest tap not being bridged onto a network where a real IMDS is routable. There is no provider-conformance test in the repo asserting that.
   Note on accessTokenHash: MMDS carries the hash of the access token, not the token, and it gates /init authorization. A workload reading its own sandbox's hash cannot forge /init with it, and it's reading metadata about the sandbox it already fully controls — so this is not a cross-boundary escalation.

3. What is the supported, testable isolation boundary for cloud-host metadata credentials? Documented protected paths / redirect handling / provider conformance tests?

   The testable boundary is DeniedSandboxCIDRs (the always-denied internal + link-local ranges, incl. 169.254.0.0/16) , enforced by the host-netns TCP proxy (tcpfirewall isIPInAlwaysDeniedCIDRs) and, for BYOP endpoints, by ValidateEgressProxy. Redirect/DNS-rebinding handling: the proxy's pre-connect() ControlContext re-checks the resolved IP and blocks a hostname that resolves into the denied ranges before the TCP handshake. There is no dedicated provider-conformance test suite or a documented "protected metadata paths" list in the repo today; coverage is CIDR/domain matching and BYOP validation tests.

Summary: not a regression or a host-credential exposure — MMDS is a Firecracker per-VM service off the filtered path, exposing only that sandbox's own metadata. The legitimate hardening opportunity is that MMDS is readable by every process in the guest (including
accessTokenHash), with no supported way to restrict it to envd without breaking init. Happy to work that as a follow-up if the team agrees on the shape (e.g. envd reads MMDS early, then drops the guest RETURN rule / firewalls 169.254.169.254:80 to non-envd processes).