Collect the last job logs before killing the container - #4289
Merged
Merged
Conversation
The server pulled logs only from `_process_running`, which runs while the job is RUNNING. Once the job started terminating, `jobs_terminating` asked the runner to stop and killed the container ten seconds later without ever pulling again, so everything the runner had buffered since the last RUNNING pull died with it -- the job's own output as it shut down, and the runner logs describing the termination. The runner has always expected that last pull. It reports `has_more` to say whether it still holds logs, `/api/stop` deliberately leaves the state non-final so the real outcome can still be handed over, and the runner then waits up to five minutes for a pull that never came. The terminating pipeline now collects what is left before stopping the container, resuming from `runner_timestamp` the same way the RUNNING pull does, and stops when the runner reports no more logs or when the deadline passes. `remove_at` becomes that deadline, which is what its name and its original docstring always said it was. It is no longer a fetcher condition: the job has to be processed during the wait rather than withheld from the pipeline, and how long to wait before killing the container belongs in `_process_terminating_job`. Jobs that never started running have no logs to wait for and are terminated as before. `running_at` is the direct signal but only exists on jobs submitted since it was added, so `runner_timestamp` stands in for jobs that were already running before the upgrade. Fixes: #4285 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The server pulled logs only from
_process_running, which runs while the job is RUNNING. Once the job started terminating,jobs_terminatingasked the runner to stop and killed the container ten seconds later without ever pulling again, so everything the runner had buffered since the last RUNNING pull died with it -- the job's own output as it shut down, and the runner logs describing the termination.The runner has always expected that last pull. It reports
has_moreto say whether it still holds logs,/api/stopdeliberately leaves the state non-final so the real outcome can still be handed over, and the runner then waits up to five minutes for a pull that never came.The terminating pipeline now collects what is left before stopping the container, resuming from
runner_timestampthe same way the RUNNING pull does, and stops when the runner reports no more logs or when the deadline passes.remove_atbecomes that deadline, which is what its name and its original docstring always said it was. It is no longer a fetcher condition: the job has to be processed during the wait rather than withheld from the pipeline, and how long to wait before killing the container belongs in_process_terminating_job.Jobs that never started running have no logs to wait for and are terminated as before.
running_atis the direct signal but only exists on jobs submitted since it was added, sorunner_timestampstands in for jobs that were already running before the upgrade.Fixes: #4285