feat(attach): support multiple concurrent attach sessions by ekalinin · Pull Request #5168 · containerd/nerdctl

ekalinin · 2026-09-01T09:06:11Z

Fixes #4374. Design discussion: #3570 (comment) and the follow-up on the switch.

This was five stacked pull requests; per review feedback on the first one it is now a single PR with the history kept separate. Rebased on current main.

The problem

A container's stdio FIFOs cannot be shared. A FIFO has one queue, so two attach sessions on the same container each receive a random subset of its output: one terminal appears to freeze while the other prints characters it never asked for. And a container started by nerdctl run -d cannot be attached to at all, because its output already went to the logging process.

dockerd does not have this problem because it owns the container's stdio and fans it out. nerdctl is daemonless, so some other process has to own it.

The approach

The owner is the internal logging process, the one nerdctl already runs as _NERDCTL_INTERNAL_LOGGING. containerd spawns it from the container's log URI, so it exists for the container's whole lifetime and comes back on its own when the restart monitor recreates a task. It gains a unix socket, a fan-out of the container's raw output tapped before the log driver's line splitting, and the write end of a stable stdin FIFO that every session's input is merged into. nerdctl attach, run -it and start -a become clients of it.

docs/dev/attach.md is the long version.

Commits


`feat(attachmux)` ×4, `fix(attachmux)` ×4	the new `pkg/attachmux`: frame protocol, broker, client, unix socket transport. Touches no existing file
`feat(cioutil)`	stable stdin FIFO at `<dataStore>/containers/<ns>/<id>/stdin.fifo`
`refactor(logging)`	tee the container's raw output before the log driver's line splitting
`feat(logging)`	start the broker inside the logging process; new leaf package `pkg/logging/loguri`
`feat(taskutil)`	declare the stable stdin FIFO on the task; `--log-driver none` goes through the internal logger
`fix(logging)` ×3	listener ordering, a Windows log URI path, and keeping the `none` driver off the buffered channels
`feat(config)`	`disable_attach_broker` as a way back to the old stdio
`feat(attach)`	`nerdctl attach` dials the socket; several sessions at once
`feat(run)`	foreground `run -it` streams through the broker
`docs(attach)`	command reference, command help, `docs/dev/attach.md`

Invariants, each with a test

The container never blocks on a session. Every session gets a byte-bounded queue and its own writer goroutine; one that falls behind is disconnected. The goroutine feeding the broker never waits on a consumer. This is the structural fix for the deadlock class in nerdctl run with attached stdio stops draining container stdout at ~72 KiB, deadlocking the container (nerdctl 2.3.5, containerd 2.3.3, Kata runtime-rs) #5137 / Fix foreground stdio deadlock when the internal logging process stops consuming #5151.
With no sessions attached the broker keeps draining, so detaching leaves the container running normally.
An exit is only announced when there was one. <-exitCh is not on its own evidence: containerd delivers Wait RPC failures on the same channel as a synthetic ExitStatus, which is what nerdctl run with attached stdio stops draining container stdout at ~72 KiB, deadlocking the container (nerdctl 2.3.5, containerd 2.3.3, Kata runtime-rs) #5137 and Fix foreground stdio deadlock when the internal logging process stops consuming #5151 are about. Rebased on Fix foreground stdio deadlock when the internal logging process stops consuming #5151, containerExited is closed after its retry loop, which is reached only by the break on a clean status - the two returns inside the loop leave the goroutine without going past it.
The exit frame does not travel through the session queue. The writer delivers it after the queue drains, so a session whose queue is exactly full still learns the container exited rather than seeing a bare EOF.
Nothing is recorded on disk about who owns a container's stdio. attach asks containerd, through the cio.Attach callback that container.Task hands the task's recorded Stdin/Stdout/Stderr. It cannot be cached: the restart monitor recreates a task from the stored log URI without nerdctl running at all, so a container that had FIFO stdio can come back with URI stdio between two invocations.

Compatibility

A container created by an older nerdctl, or on a platform where the socket is not implemented, keeps its FIFO stdio and attaches the way it always did, one session at a time. The classification and the IO construction happen in the same container.Task call, so the restart monitor cannot swap one for the other in between.

disable_attach_broker (--disable-attach-broker, NERDCTL_DISABLE_ATTACH_BROKER, nerdctl.toml) is the way back. It leaves the data store empty at task creation, which is what a container with a foreign log driver already looks like, so every branch takes the legacy path with no new code. Off by default. It only affects containers created while it is set: a running container's stdio is already bound.

The transport is built for linux || freebsd, matching where the logging process is built. Elsewhere Probe fails and run -it picks the legacy FIFO path, which is exactly today's behaviour.

Known limitations, stated in `docs/dev/attach.md`

Multi-session stdin is limited to terminal containers. For a non-terminal one the shim's binary scheme sets pio.copy = false and binaryIO.Stdin() returns nil, so stdin is not wired at all.
After a restart-policy restart, sessions are output only. The restart monitor recreates a terminal task through cio.TerminalLogURI, whose config carries no Stdin, so the shim never opens the read end of the stable FIFO. Not a regression: such a container has no stdin today either. Closing it needs the restart monitor to preserve the full IO config, which is an upstream containerd change.

Testing

go test -race ./pkg/...: green, including 53 tests in pkg/attachmux and the new cases in pkg/logging.

go vet and golangci-lint clean for linux, windows, freebsd and darwin.

New integration tests in cmd/nerdctl/container/container_attach_linux_test.go: TestAttachMultipleSessions, TestAttachToDetachedContainer, TestAttachToDetachedContainerWithoutTTY, TestAttachAfterDetachKeepsContainerResponsive.

The switch was exercised on a built binary: default off, env on, toml on, env beating toml, flag beating both.

AkihiroSuda · 2026-09-08T07:46:14Z

This PR adds the pkg/attachmux package and nothing else.

Thanks, but hard to review/test/merge until the actual nerdctl attach is implemented in the PR

ekalinin · 2026-09-12T09:35:58Z

Done - this PR is now the whole change, rebased on current main. nerdctl attach, nerdctl run -it and nerdctl start -a all go through it, so it can be tested end to end.

The five steps are kept as separate commits so the history is still reviewable in order: pkg/attachmux on its own first (8 commits, no existing file touched, no containerd needed to review it), then the logging process becoming the owner of the container's stdio, then the switch, then the two commands, then the docs.

The part I would most like a second opinion on is still the one from the issue: the internal logging process as the home for the broker, and <dataStore>/attach/<hash>.sock as the socket location. Everything else follows from those two.

Two things worth knowing before testing:

disable_attach_broker (--disable-attach-broker, NERDCTL_DISABLE_ATTACH_BROKER, nerdctl.toml) is the way back to the old stdio. Off by default, and it only affects containers created while it is set - a running container's stdio is already bound.
The transport is linux || freebsd. Elsewhere Probe fails and run -it takes the legacy FIFO path, which is today's behaviour.

There is one overlap I could not resolve on my own: #5151 touches the same task.Wait result in run. This PR only announces an exit to sessions when status.Error() == nil, which is the same distinction #5151 is making, so the two should compose - but whichever lands second will need a rebase, and I am happy to be the one to do it.

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

A failed write to the container's stdin returned from readLoop, whose deferred dropSession took the whole session with it. A container that closed its stdin and kept printing lost its terminal on the next keystroke, and one that had just exited was reported as a broken session instead of a clean exit. Also from the same review pass: - Write splits a chunk larger than one frame instead of dropping it, and refuses the control stream, which is the broker's own. - EncodeFrame moved out of b.mu. - Dial stops retrying a refused connection, which unlike a missing socket means nothing is listening. - Session.Close is no longer reported by Stream as a lost connection. - The stdin pump stops forwarding once Stream returns, so it no longer swallows the next keystroke. - Serve's context watchdog no longer leaks when Serve returns first. - Probe trims an over-long name instead of probing a longer path than a real socket. - ErrUnsupported wraps errors.ErrUnsupported. - SocketPath tests no longer sit behind a build tag, and the unsupported stubs are covered. - RemoveSocket moves here from the logging step: it is package API. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

…dropped Four decisions from the review that are cheapest to settle while the package still has no consumers. A session's queue is bounded by bytes rather than frames. A TTY echoes single keystrokes, so a frame count evicted a session that was a couple of kilobytes behind, while a frame can be up to maxPayload, so a frame count also put no real ceiling on memory in the logging process. An evicted session is told so with a control frame before it is disconnected. Otherwise falling behind is indistinguishable from the broker dying, and the client reports both as a lost connection. Listen removes the socket against its inode instead of its path. Two brokers can briefly overlap for one container while the restart monitor swaps tasks; the new one taking the path over is correct, but the old one exiting afterwards would unlink the live socket and leave the new broker on an inode nobody can reach. A client now refuses only a broker newer than itself. A broker lives as long as its container, so a client that is newer is the normal case after an upgrade, and it has nowhere to fall back to. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

The assertion spelled the separator out, so it failed on windows once the test moved out from behind the build tag. That move was the point: these cases never needed a socket, and windows is where the package is only the unsupported stubs. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

Dial guessed at the situation from the error it got, and guessed wrong in both directions. A refused connection gave up after a fixed grace period counted from the first attempt, so a container being restarted over a socket left by a killed broker fell back to the legacy path before the new broker had bound. A missing socket, meanwhile, was retried for the full five seconds, so attaching to a container that never had a broker stalled the CLI for five seconds before it could fall back. Which of those is a race and which is proof depends on the caller, not on the errno. DialStarting is for a caller that has just created the task, where containerd has not waited for the logging process it spawned and nothing is conclusive yet; Dial is for reaching a container that has been running, where a socket that is missing or refuses says there is no broker. Two more from the same review pass: - writeFrame checked for sessions only after encoding the frame, so every container with nobody attached paid for a copy of each chunk of its output, which is most containers on a host. - Listen took unlinking away from Go before it had the inode, so a failure to stat left the socket on disk with nobody to remove it. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

Two follow-ups to the attachmux review that land on this side. stop() closed the broker before the listener, so a session connecting in between was greeted by a broker that was already closed: its connection was dropped without a hello, and it had no way to tell that from a broken broker. Closing the listener first shuts that window. RemoveContainer now removes the container's attach socket. It lives outside the container's state directory, because sun_path is too short to hold that path, so removing the state directory did not take it with it. A broker that exited normally unlinked it already; one that was killed did not, and nothing else would ever collect it. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

Multi-session attach changes how a container's stdio is wired, and there is no way back once a task exists: its stdout is a binary:// URI, which cio.NewAttach cannot open. So the choice has to be made when the task is created, and an operator has to be able to make it. --disable-attach-broker, NERDCTL_DISABLE_ATTACH_BROKER and disable_attach_broker in nerdctl.toml all reach the same switch, through the same helper the other global options use. Off is the default. The name is negative on purpose. This is not a feature switch but a way to back the change out in the field without a downgrade, and it is expected to go away once the broker has proven itself. It leaves the data store empty at task creation, which every path already handles: that is also what a foreign log driver looks like. --log-driver none short-circuits again as well, so a container that asked for no logs runs without a logging process at all, as it did before. Containers already running are not affected: their stdio is bound. Backing the change out means switching it off and restarting them. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

Three failures the fork's CI found on the second step, which the first step could not have surfaced. cio.LogURIGenerator always gives the path a leading slash, so that a windows path is not mistaken for a host name. IsInternal passed that straight to filepath.EvalSymlinks, which on windows cannot resolve "/C:/...", so a container's own log URI was not recognised as internal. Harmless today, since the transport is not built for windows and every caller falls back, but it would have broken quietly once it is. The trim is now in BinaryPath, next to the same trim taskutil already does. TestDataStore built a URI from a hardcoded POSIX path, which LogURIGenerator rejects on windows for not being absolute there. addRootFlagsForConvertOptionsTest mirrors every root flag that ProcessRootCmdFlags reads, so adding a global option means adding it there too. Missing it made every linux integration suite fail on a single image test. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

Three ways the second step broke behaviour that works today. --log-driver none now goes through the logging process, because that is where the attach broker lives. NoneLogger.Process returned without ever reading its channels, and NoneLogger was not a SyncDriver, so the logger queued every line for a consumer that never read: past the buffer the container blocked writing to its own stdout. It implements SyncDriver now, which keeps it off the channels entirely. The stable stdin FIFO is created only for -it, not for any -i. The broker holds a second writer on that FIFO for the container's lifetime, which is what makes detaching leave a terminal running, but it also means a non-terminal container never sees EOF on its stdin: task.CloseIO closes the shim's writer and nerdctl's, not the broker's, so "echo x | nerdctl run -i alpine cat" would have hung. Multi-session stdin was limited to terminal containers anyway. This also stops "run -d -i" without -t from creating a FIFO the shim never opens, which cost 30 seconds of retried opens per container. RemoveContainer removes the attach socket after the container is gone, not before it is checked. Removing it first meant a plain "nerdctl rm" of a running container, which stops at the status check, had already unlinked the socket of a container that keeps running, leaving its broker on an inode nobody can reach. Whether the container has a terminal is read from log-config.json, which the CLI already writes and the logging process already reads, rather than from containerd. Asking containerd put a connect and two RPCs on the critical path of creating the task, since the shim blocks on ready(). Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

nerdctl attach now asks containerd how the task's stdio is wired and, for a container whose output goes to the internal logging process, streams over that process's attach socket instead of re-opening the FIFOs. Reading it from containerd rather than from anything nerdctl recorded is what makes this correct across a restart: the restart monitor recreates a task straight from the stored log URI, without nerdctl running, so a container that had FIFO stdio can come back with URI stdio between two invocations. The classification and the IO construction happen in the same container.Task call, so the restart monitor cannot change the answer in between either. Containers whose stdio is still a set of FIFOs keep today's path, one session at a time. TestAttachMultipleSessions is the reproduction from containerd#4374: two sessions typing into one container, both of which must see all of its output. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

run -it and start -a hand the container's stdio to the logging process and become clients of its socket, the same way attach now is. That closes the second half of the problem: until now a foreground session drained the FIFOs itself, so detaching from it left nobody reading them and the container stalled once the pipe buffer filled. The decision has to be made before the task exists, since a task built for the broker cannot be attached to any other way, so run and start probe the socket directory first and fall back to the FIFO path when it is unavailable. If the socket turns out to be unreachable afterwards the task is deleted before it starts, rather than leaving a container whose output nobody can show. The ioCreator chain in taskutil becomes a switch. It has six branches now and the order between them carries meaning: the broker case has to come before the AttachStreamOpt case, which start -a sets, while the detached terminal case has to come after it. As an if/else-if chain it also tipped over the nesting limit in .golangci.yml. terminalBrokerIO uses loguri.BinaryPath rather than repeating the leading slash trim that a binary log URI needs on windows. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

The caveat about a single attach session, and the one about a detached container not being attachable at all, no longer describe what happens. Replace them with what is true now, on the command and in the command reference, and add docs/dev/attach.md for the design: why the logging process owns the stdio, what the broker guarantees, why stdin is limited to terminal containers, what a restart-policy restart loses, and how a container that predates the socket still attaches. Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

ekalinin · 2026-09-18T07:58:01Z

Rebased on current main, conflicts resolved - #5151 landed in the meantime, which is the overlap I mentioned above.

ekalinin marked this pull request as ready for review September 2, 2026 12:31

This was referenced Sep 2, 2026

Multiple attach should be supported #3570

Open

feat(logging): serve attach sessions from the container logging process ekalinin/nerdctl#1

Closed

AkihiroSuda added this to the v2.4.1 milestone Sep 8, 2026

ekalinin force-pushed the feat/attach-multiplexer branch from 310a28e to 50687c3 Compare September 12, 2026 09:35

ekalinin changed the title ~~feat(attachmux): add the stdio multiplexing package for multi-session attach~~ feat(attach): support multiple concurrent attach sessions Sep 12, 2026

ekalinin added 19 commits September 18, 2026 10:13

feat(attachmux): add stdio multiplexing frame protocol

ed66171

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

feat(attachmux): add the container stdio broker

8ad9f36

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

feat(attachmux): add the client attach session

081e399

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

feat(attachmux): add the attach socket transport

0c04358

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

feat(cioutil): add a stable per-container stdin FIFO path

1406961

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

refactor(logging): tee raw container output before line splitting

7dbddf7

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

feat(logging): serve attach sessions from the container logging process

b5e96ff

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

feat(taskutil): pin container stdin to a stable FIFO path

41123b4

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>

ekalinin force-pushed the feat/attach-multiplexer branch from 50687c3 to 37dfaa4 Compare September 18, 2026 07:57

Navigation Menu

feat(attach): support multiple concurrent attach sessions - #5168

feat(attach): support multiple concurrent attach sessions#5168
ekalinin wants to merge 19 commits into
containerd:mainfrom
ekalinin:feat/attach-multiplexer

ekalinin commented Sep 1, 2026 •

edited

Loading

Uh oh!

AkihiroSuda commented Sep 8, 2026

Uh oh!

ekalinin commented Sep 12, 2026 •

edited

Loading

Uh oh!

ekalinin commented Sep 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ekalinin commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

The approach

Commits

Invariants, each with a test

Compatibility

Known limitations, stated in docs/dev/attach.md

Testing

Uh oh!

AkihiroSuda commented Sep 8, 2026

Uh oh!

ekalinin commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ekalinin commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ekalinin commented Sep 1, 2026 •

edited

Loading

Known limitations, stated in `docs/dev/attach.md`

ekalinin commented Sep 12, 2026 •

edited

Loading

ekalinin commented Sep 18, 2026 •

edited

Loading