Skip to content

fix(kill): do not mark non-stop signals as an explicit stop - #5174

Open
somaz94 wants to merge 1 commit into
containerd:mainfrom
somaz94:fix/kill-non-stop-signal-label
Open

somaz94 wants to merge 1 commit into
containerd:mainfrom
somaz94:fix/kill-non-stop-signal-label

Conversation

@somaz94

@somaz94 somaz94 commented Sep 2, 2026

Copy link
Copy Markdown

nerdctl kill set containerd.io/restart.explicitly-stopped before sending
any signal, so nerdctl kill --signal HUP on a --restart=always container
permanently suppressed its restart policy. Nothing reported it: the container
kept running and looked healthy.

The label is now set only when the signal is expected to stop the container:
SIGKILL, or a signal matching the container's own stop signal. That stop
signal is resolved by the same rule nerdctl stop already uses
(containerutil.getSignal): the stopSignal label if present, SIGTERM
otherwise, so the two commands cannot disagree about what stops a container.
An unparsable stopSignal label is returned as an error rather than guessed
at, matching what nerdctl stop already does with it.

nerdctl stop is unchanged: it stops the container by definition, so it
still sets the label unconditionally.

Validation:

  • make lint-go-all (linux/windows/freebsd/darwin) and make lint-mod: 0 issues,
    using the golangci-lint version pinned in install-dev-tools
  • make lint-commits: PASS
  • New unit test TestIsStopSignal (6 cases) and integration test
    TestRunRestartAlwaysKillSignal (3 subtests) pass against containerd 2.2.2
    • runc 1.4.0 on Ubuntu 26.04 (aarch64)
  • Checked both tests actually fail without the fix: making IsStopSignal
    always return true fails only the non-stop-signal subtest, and narrowing it
    to SIGKILL only fails only the stop-signal-match subtest
  • make test: unchanged except pkg/resolvconf.TestGet, which fails the same
    way on a clean checkout of main in this environment

closes: #5171

somaz94 added a commit to somaz94/somaz94 that referenced this pull request Sep 2, 2026
@somaz94
somaz94 marked this pull request as ready for review September 2, 2026 06:00
@somaz94

somaz94 commented Sep 2, 2026

Copy link
Copy Markdown
Author

The three failing jobs look unrelated to this change.

TestSaveQuiet, TestLoadQuiet and TestSaveContent are image save/load tests,
and TestNetworkInspectWithContainers fails on hcnCreateNetwork "object already
exists". This PR only touches killContainer. I ran all three save/load tests
against this branch on containerd 2.2.2 and they pass.

The test workflow on main has failed 14 of its last 15 runs, with different
tests each time, so I don't think waiting for a green run here is meaningful.
Happy to rebase if you'd rather I retry against a newer main.

// This asserts the label nerdctl itself writes, so it does not depend on the
// containerd restart monitor actually acting on it. It is containerd-only
// because docker has no equivalent label.
testCase.Require = require.Not(nerdtest.Docker)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than checking the internal label, this test should just ensure that the restart manager is working in the same way as Docker when a non-stop signal is sent?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rewritten as suggested.

The test now asserts the restart behaviour rather than the label, so the docker gate is gone and it runs against both. One case sends SIGHUP, which is not the stop signal, then kills the container process directly, and expects the restart policy to bring it back with a restart count of 1. The other sends the default SIGKILL and expects the container to stay down with a restart count of 0.

Worth noting for the first case: sleep runs as PID 1 and installs no handler, so the kernel discards SIGHUP and the container keeps running. Killing the process afterwards is what makes it exit for a reason that is not a user stop, which is the condition the fix affects.

The red rootless job looks unrelated. It failed here on TestSaveMultipleImagesWithSameIDAndLoad, while the same job on main fails on different tests (TestLoadStdinFromPipe, TestRunRmTime), and 4 of the last 8 main runs have failing rootless jobs.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have to correct my previous reply. I reverted the behavioural rewrite and put the label assertions back, because measuring it showed the behavioural version does not test anything.

Setup was containerd 2.2.2 on Ubuntu aarch64, comparing a build of this branch against the same build with IsStopSignal forced to always return true, which is the pre-fix behaviour. The behavioural test passes in both. After kill --signal HUP and then killing the container process, the container comes back with a restart count of 1 whether or not the label was set, and I got the same result across a containerd restart. So explicitly-stopped does not change the observable restart outcome in these scenarios, and a test written against that outcome cannot tell the fix from its absence.

The label assertions do discriminate. With the fix all three subtests pass; with the pre-fix build only the non stop signal subtest fails. That is why they are back.

Separately I was wrong to say the test could run against docker. In docker 29.4 any docker kill suppresses the restart policy regardless of signal: a --restart=always container that exits on its own restarts normally, but after docker kill --signal HUP or --signal USR1 it stays exited with a restart count of 0. In daemon/kill.go, ExitOnNext is conditional on the stop signal matching, but HasBeenManuallyStopped = true a few lines below is not.

That last point cuts against the framing here, so it is your call rather than mine. If matching docker is the goal, the pre-fix behaviour was already the matching one and #5171 is a docker compatible quirk. If the goal is nerdctl's own consistency, that a running container is silently made ineligible for restart, the change stands and the label assertion is the only check I found that verifies it.

@somaz94
somaz94 force-pushed the fix/kill-non-stop-signal-label branch from 92b50ae to cfc6906 Compare September 10, 2026 02:53
func TestRunRestartAfterKillSignal(t *testing.T) {
testCase := nerdtest.Setup()
if !nerdtest.IsDocker() {
testCase.Require = nerdtest.ContainerdPlugin("io.containerd.internal.v1", "restart", []string{"always"})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is already resolved upstream. f99df631 dropped the containerd 1.x
plugin requirement from these restart tests, and #5175 is closed. The test
added here never calls nerdtest.ContainerdPlugin at all -- its only Require
is require.Not(nerdtest.Docker), which is why the thread shows as outdated.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please squash the commits

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Squashed -- the branch is a single commit (4baa329a) on top of main.

Comment thread pkg/containerutil/containerutil_test.go Outdated
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got != tt.expected {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use gotest.tools/v3/assert

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, switched to gotest.tools/v3/assert.

The error case is now assert.Assert(t, err != nil), and the happy path is
assert.NilError(t, err) plus assert.Equal(t, got, tt.expected). The import
sits in the default gci group, matching pkg/imgutil/snapshotter_test.go.
I left the pre-existing tests in this file alone to keep the diff scoped.

@somaz94
somaz94 force-pushed the fix/kill-non-stop-signal-label branch from cfc6906 to 86c0e92 Compare September 10, 2026 03:26
`nerdctl kill` accepts any signal, but it recorded every one of them by
setting containerd.io/restart.explicitly-stopped to true before sending it.
That label is what tells the containerd restart monitor to leave a container
alone, so sending a signal that is not meant to stop the container, such as
SIGHUP, permanently suppressed its restart policy. The container kept running
and looked healthy, and nothing reported that its --restart=always had stopped
applying.

Set the label only when the signal is expected to stop the container: SIGKILL,
or a signal matching the container's own stop signal. The stop signal is
resolved by the same rule `nerdctl stop` already uses, the stopSignal label if
present and SIGTERM otherwise, so the two commands cannot disagree about what
stops a container. An unparsable stopSignal label is returned as an error
rather than guessed at, which is what `nerdctl stop` already does with it.

`nerdctl stop` is unchanged. It stops the container by definition, so it still
sets the label unconditionally.

Assisted-by: Claude Code (Opus 5)
Signed-off-by: somaz <genius5711@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nerdctl kill shouldn't set containerd.io/restart.explicitly-stopped label for non-stop signals

2 participants