ateom-microvm: upgrade cloud-hypervisor to v53 and boot directly to agent - #972
Open
Benjamin Elder (BenTheElder) wants to merge 2 commits into
Open
ateom-microvm: upgrade cloud-hypervisor to v53 and boot directly to agent#972Benjamin Elder (BenTheElder) wants to merge 2 commits into
Benjamin Elder (BenTheElder) wants to merge 2 commits into
Conversation
cloud-hypervisor publishes minor releases and rarely backports to a patch branch, so staying on v52.0 means running a hypervisor that receives nothing. v53.0 needs the restore mode chosen for it, which the preceding commits do: its userfaultfd restore handler background-prefaults every registered page, which starves the guest badly enough that an actor never passes its readiness probe. Restoring eagerly avoids that, and the runtime now selects it from the version the VMM reports. The cost is memory: a restored idle actor holds its whole snapshot resident rather than the pages it touches, ~158MiB against ~16MiB on the counter demo, because prefaulting fills whatever is registered. Nothing configurable recovers that; the ways out are a smaller guest working set or in-VMM snapshot chaining upstream (cloud-hypervisor#8645). Suspend cost and per-actor disk are unchanged, at ~0.32s and ~158MiB.
The guest ran a full Ubuntu systemd to start one process. Measured on the
counter demo, systemd and its library closure were ~40MiB of the ~68MiB the
guest read from its rootfs disk at boot; the agent itself links only libc,
libm, libgcc_s and ld.so.
Boot it directly with init=/usr/bin/kata-agent. The agent already detects
PID 1 and does the init work (mounts /proc, /sys, devtmpfs /dev, /dev/shm,
/dev/pts, tmpfs /run, cgroups, sets the hostname), which is how kata's own
initrd variant runs it.
Measured on kind (arm64, counter demo, 1280MiB guest, same base and template,
three golden bakes per arm):
snapshot 145MiB -> 106.6MiB (-26%)
read from /dev/vda 58.6MiB -> 35.0MiB (-40%)
guest page cache 63MiB -> 40MiB
cold boot 15924ms -> 10308ms (-35%)
of which the agent dial 10361ms -> 4675ms (-55%)
guest userspace systemd + journald + 2x chronyd + agent + workload
-> agent + workload
The boot win is the dial phase: the agent is PID 1 rather than a unit systemd
reaches several seconds into its startup, so ateom stops waiting on it. It is
also much steadier - 10.298/10.310/10.316s across three bakes, against
14.4-17.9s for the same tree with systemd.
Gate it on the VMM version, because dropping systemd drops chronyd with it
(kata-containers.target wants it) and chronyd is what repaired the guest clock
across a resume. cloud-hypervisor v53 advances the guest clock on restore
itself; v52 does not, so an older or unreadable version keeps booting systemd.
Verified with chronyd masked and 150s of downtime: a v53 guest resumes reading
the correct wall clock, a v52 guest is 160s behind, frozen at the pause. The
version comes from the vmm.ping LaunchVMM already does, so the gate costs no
extra round trip.
Benjamin Elder (BenTheElder)
force-pushed
the
microvm-v53-agent-init
branch
from
August 15, 2026 05:50
07b9e30 to
fefbc5f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #847
cloud-hypervisor v53 ships guest clock updates:
https://github.com/cloud-hypervisor/cloud-hypervisor/releases#:~:text=Guest%20Clock%20Updates%20Across%20Snapshot/Restore%20and%20Migration
In v52, we've been booting to systemd => {chrony, journald, kata-agent}
We don't really have much use for journald, as we get logs from containers already.
chrony however was load-bearing to re-sync the clock on resume, and therefore systemd was load-bearing (to run chronyd + kata-agent).
We don't need chrony anymore (tested with long suspend and probing the reported time in the actor), so we can boot directly to the agent.
This buys us some startup time, and it drops the memory floor by ~40MB. On golden for counter demo this is ~-26%.
Reducing memory snapshot size speeds up snapshot/restore.