Skip to content

Collect the last job logs before killing the container - #4289

Merged
un-def merged 1 commit into
masterfrom
issue_4285_collect_last_job_logs
Sep 15, 2026
Merged

un-def merged 1 commit into
masterfrom
issue_4285_collect_last_job_logs

Conversation

@un-def

@un-def un-def commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

The server pulled logs only from _process_running, which runs while the job is RUNNING. Once the job started terminating, jobs_terminating asked the runner to stop and killed the container ten seconds later without ever pulling again, so everything the runner had buffered since the last RUNNING pull died with it -- the job's own output as it shut down, and the runner logs describing the termination.

The runner has always expected that last pull. It reports has_more to say whether it still holds logs, /api/stop deliberately leaves the state non-final so the real outcome can still be handed over, and the runner then waits up to five minutes for a pull that never came.

The terminating pipeline now collects what is left before stopping the container, resuming from runner_timestamp the same way the RUNNING pull does, and stops when the runner reports no more logs or when the deadline passes.

remove_at becomes that deadline, which is what its name and its original docstring always said it was. It is no longer a fetcher condition: the job has to be processed during the wait rather than withheld from the pipeline, and how long to wait before killing the container belongs in _process_terminating_job.

Jobs that never started running have no logs to wait for and are terminated as before. running_at is the direct signal but only exists on jobs submitted since it was added, so runner_timestamp stands in for jobs that were already running before the upgrade.

Fixes: #4285

The server pulled logs only from `_process_running`, which runs while
the job is RUNNING. Once the job started terminating, `jobs_terminating`
asked the runner to stop and killed the container ten seconds later
without ever pulling again, so everything the runner had buffered since
the last RUNNING pull died with it -- the job's own output as it shut
down, and the runner logs describing the termination.

The runner has always expected that last pull. It reports `has_more` to
say whether it still holds logs, `/api/stop` deliberately leaves the
state non-final so the real outcome can still be handed over, and the
runner then waits up to five minutes for a pull that never came.

The terminating pipeline now collects what is left before stopping the
container, resuming from `runner_timestamp` the same way the RUNNING
pull does, and stops when the runner reports no more logs or when the
deadline passes.

`remove_at` becomes that deadline, which is what its name and its
original docstring always said it was. It is no longer a fetcher
condition: the job has to be processed during the wait rather than
withheld from the pipeline, and how long to wait before killing the
container belongs in `_process_terminating_job`.

Jobs that never started running have no logs to wait for and are
terminated as before. `running_at` is the direct signal but only exists
on jobs submitted since it was added, so `runner_timestamp` stands in
for jobs that were already running before the upgrade.

Fixes: #4285
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@un-def
un-def merged commit 979ac05 into master Sep 15, 2026
26 checks passed
@un-def
un-def deleted the issue_4285_collect_last_job_logs branch September 15, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Job log tail lost on dstack stop

1 participant