Skip to content

fix(envd): mount sandbox volumes with nolock to stop flock hang (#3619) - #3623

Open
AdaAibaby wants to merge 2 commits into
e2b-dev:mainfrom
AdaAibaby:fix/nfs-volume-nolock-flock-hang-3619
Open

fix(envd): mount sandbox volumes with nolock to stop flock hang (#3619)#3623
AdaAibaby wants to merge 2 commits into
e2b-dev:mainfrom
AdaAibaby:fix/nfs-volume-nolock-flock-hang-3619

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Summary

flock() on a mounted sandbox volume hangs forever in uninterruptible sleep (state D), cannot be killed even with SIGKILL, and holds the sandbox until it is destroyed. Reads and writes are fine; only locking hangs.

Root cause: nfsOptions in packages/envd/internal/api/init.go sets neither nolock nor local_lock, so local_lock defaults to none and the kernel sends every advisory lock to the nfs proxy as an NLM request. The proxy has no lock manager and its portmap (packages/orchestrator/pkg/portmap) never registers NLM (100021), so PMAPPROC_GETPORT returns port 0. Because the mount is hard, the client retries the bind forever.

Fix: add nolock to nfsOptions. Locks then resolve node-locally, which is the correct semantics given the proxy has no lock manager — and matches how the same server is mounted everywhere else in the repo (nfsproxy e2e nolock, Filestore chunk cache nolock).

Changes

  • packages/envd/internal/api/init.go — add nolock to nfsOptions (the fix).
  • packages/envd/internal/services/cgroups/hierarchy.go — the pause/resume freeze allowlist documented its rpcbind/rpc-statd entries as justified by "mounts have no nolock, so mount.nfs starts rpc.statd". With nolock, mount.nfs no longer starts rpc.statd, so that justification is gone. Comment updated to reflect reality; the entries are kept defensively (re-justified inline) rather than pruned, to keep this PR's blast radius on freeze behavior at zero.
  • packages/orchestrator/pkg/nfsproxy/e2e_test.sh — add an flock assertion (10s timeout) to the POSIX suite so a future regression that drops nolock fails loudly instead of hanging the suite. This closes the gap that let the bug ship: the e2e harness (e2e_start.sh) mounts with nolock, so it never exercised the production config.

Why this is not a duplicate

Checked before opening:

  • gh issue view 3619 --repo e2b-dev/infra --comments — no comments, no linked PR.
  • gh pr list --state open --search '3619 in:body' — none.
  • gh pr list --state open --search 'nolock' / 'flock' / 'nfsOptions' — no PR touching this.

Tests run

Reproduced and verified on a dev host using the real proxy + portmap code (unmodified), two containers on a bridge network mirroring production topology (client with local rpcbind+statd; server = go-nfs proxy+portmap registering only NFS/mountd on 2049, never NLM):

Before (current prod options, no nolock) — mount reports local_lock=none; flock on the volume hangs, killed by timeout (exit 124). Kernel evidence matches the issue exactly:

state D, wchan rpc_wait_bit_killable
rpc_wait_bit_killable -> __rpc_execute -> rpc_call_sync -> nlmclnt_call ->
nlmclnt_lock -> nlmclnt_proc -> nfs3_proc_lock -> do_setlk -> nfs_flock ->
__do_sys_flock -> do_syscall_64

Local-disk flock (control) and all reads/writes succeed.

After (with nolock) — same unmodified proxy; mount reports local_lock=all; flock on the volume returns 0 (LOCKED_OK).

Also:

  • go build ./... (envd) — OK.
  • go vet on internal/api, internal/services/cgroups, pkg/nfsproxy — OK.
  • gofmt -l on changed Go files — clean.
  • bash -n e2e_test.sh — OK; ran e2e_test.sh standalone, new flock assertion passes.

The full TestIntegrationTest could not run on this shared box because the host rpcbind already holds :111 (a pre-existing environment constraint, not related to this change); the flock behavior was instead proven with the isolated two-container harness above.

Model evaluation

Not applicable — no change to model output, accuracy, or serving. This only changes NFS mount options and a test/comment.

AI assistance

AI assistance was used to investigate, reproduce, and draft this change. A human submitter has reviewed every changed line and the reproduction.

Fixes #3619

Sandbox volume mounts set neither nolock nor local_lock in nfsOptions, so
local_lock defaults to none and the kernel routes every advisory lock to the
nfs proxy as an NLM request. The proxy has no lock manager and its portmap
never registers NLM (100021), so the portmapper GETPORT lookup returns port 0.
Because the mount is hard, the client retries the bind forever and the caller
(e.g. flock in $CODEX_HOME) wedges in uninterruptible sleep (state D, wchan
rpc_wait_bit_killable) until the sandbox is destroyed.

Add nolock so locks resolve node-locally, which is the correct semantics given
the proxy has no lock manager -- and matches how the same server is mounted
everywhere else (nfsproxy e2e, Filestore chunk cache).

As a side effect mount.nfs no longer starts rpc.statd during setupNFS, so the
pause/resume freeze allowlist in cgroups/hierarchy.go no longer depends on the
local portmapper; its comment is updated and the now-defensive rpcbind entries
are kept and re-justified.

Also add an flock assertion to the nfsproxy e2e POSIX suite so a future
regression that drops nolock fails loudly (10s timeout) instead of hanging.

Reproduced on a dev host with the real proxy + portmap: without nolock,
flock on the volume hangs (state D, exact kernel stack from the issue:
rpc_wait_bit_killable -> nlmclnt_call -> nlmclnt_lock -> nfs3_proc_lock ->
nfs_flock -> __do_sys_flock) while a local-disk flock and all reads/writes
succeed; with nolock the mount reports local_lock=all and flock returns 0.

Fixes e2b-dev#3619

Signed-off-by: AdaAibaby <shaolila@buaa.edu.cn>
@AdaAibaby
AdaAibaby force-pushed the fix/nfs-volume-nolock-flock-hang-3619 branch from 5b4439c to cebb064 Compare September 7, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: flock() on a mounted volume hangs forever (mount is missing nolock)

2 participants