Skip to content

Prevent run_cvd abort and keep monitor socket alive on graceful VM sh… - #3154

Open
SuperStrongDinosaur wants to merge 2 commits into
google:mainfrom
SuperStrongDinosaur:FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2
Open

SuperStrongDinosaur wants to merge 2 commits into
google:mainfrom
SuperStrongDinosaur:FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2

Conversation

@SuperStrongDinosaur

@SuperStrongDinosaur SuperStrongDinosaur commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

When a Cuttlefish guest VM gracefully powers off, crosvm terminates cleanly, which in turn leads the ProcessMonitor subprocess to stop all monitored processes and exit with code 0.

Previously, run_cvd treated any readability on the monitor status pipe as an not normal failure. This caused run_cvd to immediately abort and abandon launcher_monitor.sock. As a result:
1. The VM entered an Unreachable state .
2. Subsequent cvd restart, cvd stop, or cvd powerwash commands failed with Connection refused.

Solution

  1. Keep run_cvd Server Loop Alive on Exit:
    - Track process_monitor_active. When the monitor exit pipe is triggered in Select(), mark process_monitor_active = false so subsequent iterations only poll server_ .
    - Immediately invoke process_monitor.StopMonitoredProcesses() to wait for and clean up the terminated monitor child process, preventing lingering <defunct> zombie processes.
    - Keep the server loop running to service subsequent launcher actions (cvd restart, cvd stop, cvd powerwash, and cvd status).

  2. Harden StopMonitoredProcesses():
    - Make StopMonitoredProcesses() safe to call multiple times and handle cases where the monitor has already exited or finished.
    - Eliminate mid-shutdown race conditions: if SendEmptyRequest encounters a closed/shut down socket because the monitor child is already in its exit path, do not abort early. Instead, proceed to wait for the process to exit using waitpid(last_monitor, &wstatus, 0) and log abnormal exits as warnings rather than failing the stop/restart request.

• Verified lifecycle workflows:
• Graceful guest VM shutdown via VePSM / adb reboot -p leaves launcher_monitor.sock active and eliminates zombie processes.
• cvd restart successfully connects, reaps state, and re-executes run_cvd.
• cvd stop cleanly terminates run_cvd with exit code 0.
• cvd status and cvd fleet successfully respond without connection errors.

Bug: b/534717429

@SuperStrongDinosaur
SuperStrongDinosaur force-pushed the FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2 branch 3 times, most recently from e6e2de6 to 5a46719 Compare September 10, 2026 09:21
@SuperStrongDinosaur SuperStrongDinosaur added the kokoro:run Run e2e tests. label Sep 10, 2026
@GoogleCuttlefishTesterBot GoogleCuttlefishTesterBot removed the kokoro:run Run e2e tests. label Sep 10, 2026
@SuperStrongDinosaur
SuperStrongDinosaur marked this pull request as ready for review September 10, 2026 15:47
Comment on lines +131 to +132
LOG(INFO) << "Process monitor has exited (guest VM shut down). Server "
"loop continuing to listen for status/restart.";

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this assume that every process monitor exit implies a graceful VM shutdown? There's also the scenario that crosvm or another critical process exits unexpectedly and an error should be reported.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Yes, right now, run_cvd assumes any process monitor termination is a graceful shutdown, which would mask an unexpected crash of crosvm or something else.

We can differentiate between the two cases using the exit status of the monitor process:

  1. Graceful Shutdown: The crosvm exits cleanly with code 0 . In this case, ProcessMonitor exits with code 0, and run_cvd keeps the launcher socket open to listen for cvd restart/status.
  2. Unexpected Crash: A critical process exits with a signal (WIFSIGNALED) or a non-zero exit code. In this case, ProcessMonitor exits with code 1, and run_cvd reports an error .

Would you prefer updating ProcessMonitor to propagate this exit status so run_cvd can abort on unexpected crashes and only stay alive on 0 exits?

@SuperStrongDinosaur
SuperStrongDinosaur force-pushed the FixCuttlefishLauncherCrashOnGracefulGuestVMShutdown2 branch from 5a46719 to d7de57b Compare September 14, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants