atelet: stage local checkpoints by hard link instead of copying

[Jeffrey Ying (Jefftree)]

Resuming from a node-local pause checkpoint copied the whole snapshot into the restore dir, so the node held two copies of the same guest memory image and paid a full write to make the second. Both dirs sit under the same actor dir, so link instead, falling back to a copy across filesystems.

Linking makes the staged image share an inode with the cached snapshot, which MergeDeltaIntoBase would otherwise overlay in place and corrupt. Two changes keep that safe. Earlier pause snapshots are now pruned before the checkpoint rather than after, which releases the second link while the staged image still holds the inode, so the merge keeps its cheap in-place path. And the merge refuses that path outright when base still carries a second link, copying instead, as a backstop for any ordering it cannot see.

Counter demo on kind, 2 GiB guest with ~133 MiB populated:

	before	after
resume, staging the image	105 ms	under 1 ms
suspend, merge	16 ms	14 ms

The merge figure is from the Merged OnDemand delta into base log line on a live cluster across a pause, resume, pause cycle, confirming the prune lands before the merge reads the link count. Without the reordering the same merge costs ~130 ms.

[Benjamin Elder (BenTheElder)]

🤖 question 🟢 – The stated mitigation covers an RPC failure but not a crash, and the reordering widens the window where neither snapshot exists.

A gap between the prune and moveLocalCheckpoint was already there, but it used to be short. It now spans the whole CheckpointWorkload call — pause, write the memory image, tear down — which is the expensive part. Through all of it the node holds no local snapshot while the actor's LocalSnapshotInfo still names the pruned one.

"The checkpoint can simply be retried" holds when the RPC returns an error, because the guest is still paused. It doesn't hold if atelet or the node dies mid-snapshot: nothing retries, and the actor is left pinned by RequiredNodes to a node whose local snapshot has been deleted. Before this change the same crash left the earlier snapshot intact and the resume worked.

Keeping the old snapshot isn't free — the merge would take the copying path at roughly 130ms instead of 14ms — so trading that for a wider crash window may well be the right call. Worth saying so in the comment, though, since it currently reads as if a retry always covers the cost.

[Benjamin Elder (BenTheElder)]

Human: We would probably prefer to keep the disk fuller than lose reliability.

[Jeffrey Ying (Jefftree)]

Yeah this is a tradeoff losing reliability for improved performance. It's also a suspend only optimization and resume seems more important. Removed the early pruning.