Prevent run_cvd abort and keep monitor socket alive on graceful VM sh… - #3154
Open
SuperStrongDinosaur wants to merge 2 commits into
Open
SuperStrongDinosaur wants to merge 2 commits into
SuperStrongDinosaur wants to merge 2 commits into
Conversation
SuperStrongDinosaur
force-pushed
the
FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2
branch
3 times, most recently
from
September 10, 2026 09:21
e6e2de6 to
5a46719
Compare
SuperStrongDinosaur
marked this pull request as ready for review
September 10, 2026 15:47
Databean
reviewed
Sep 11, 2026
Comment on lines
+131
to
+132
| LOG(INFO) << "Process monitor has exited (guest VM shut down). Server " | ||
| "loop continuing to listen for status/restart."; |
Member
There was a problem hiding this comment.
Does this assume that every process monitor exit implies a graceful VM shutdown? There's also the scenario that crosvm or another critical process exits unexpectedly and an error should be reported.
Collaborator
Author
There was a problem hiding this comment.
I see. Yes, right now, run_cvd assumes any process monitor termination is a graceful shutdown, which would mask an unexpected crash of crosvm or something else.
We can differentiate between the two cases using the exit status of the monitor process:
- Graceful Shutdown: The
crosvmexits cleanly with code 0 . In this case,ProcessMonitorexits with code 0, andrun_cvdkeeps the launcher socket open to listen forcvd restart/status. - Unexpected Crash: A critical process exits with a signal (
WIFSIGNALED) or a non-zero exit code. In this case,ProcessMonitorexits with code 1, andrun_cvdreports an error .
Would you prefer updating ProcessMonitor to propagate this exit status so run_cvd can abort on unexpected crashes and only stay alive on 0 exits?
SuperStrongDinosaur
force-pushed
the
FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2
branch
from
September 14, 2026 16:45
5a46719 to
d7de57b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a Cuttlefish guest VM gracefully powers off,
crosvmterminates cleanly, which in turn leads theProcessMonitorsubprocess to stop all monitored processes and exit with code 0.Previously,
run_cvdtreated any readability on the monitor status pipe as an not normal failure. This causedrun_cvdto immediately abort and abandonlauncher_monitor.sock. As a result:1. The VM entered an
Unreachablestate .2. Subsequent
cvd restart,cvd stop, orcvd powerwashcommands failed withConnection refused.Solution
Keep
run_cvdServer Loop Alive on Exit:- Track
process_monitor_active. When the monitor exit pipe is triggered inSelect(), markprocess_monitor_active = falseso subsequent iterations only pollserver_.- Immediately invoke
process_monitor.StopMonitoredProcesses()to wait for and clean up the terminated monitor child process, preventing lingering<defunct>zombie processes.- Keep the server loop running to service subsequent launcher actions (
cvd restart,cvd stop,cvd powerwash, andcvd status).Harden
StopMonitoredProcesses():- Make
StopMonitoredProcesses()safe to call multiple times and handle cases where the monitor has already exited or finished.- Eliminate mid-shutdown race conditions: if
SendEmptyRequestencounters a closed/shut down socket because the monitor child is already in its exit path, do not abort early. Instead, proceed to wait for the process to exit usingwaitpid(last_monitor, &wstatus, 0)and log abnormal exits as warnings rather than failing the stop/restart request.• Verified lifecycle workflows:
• Graceful guest VM shutdown via VePSM / adb reboot -p leaves launcher_monitor.sock active and eliminates zombie processes.
• cvd restart successfully connects, reaps state, and re-executes run_cvd.
• cvd stop cleanly terminates run_cvd with exit code 0.
• cvd status and cvd fleet successfully respond without connection errors.
Bug: b/534717429