feat: add virtqueue transport foundations by andreiltd · Pull Request #1793 · hyperlight-dev/hyperlight

andreiltd · 2026-09-03T13:09:49Z

The patch implements shared layout and ownership model for virtq based host and guest communication. This is a preparation work for replacing stack based IO with virtq, in the meantime function calls and logging use the legacy transport. Actual takeover, buffer mappings, guest allocated pools and retained buffer snapshots are separate stages.

queue configurations and a shared scratch arena for rings and buffer pools.
SlotPool allocation,
stateful chain I/O,
shared message framing,
guest queue initialization,
canonical ring state validation

Copilot

🟡 Changes recommended

TransportArena::new currently sets h2g_pool_addr to the G2H pool offset, causing pool overlap and an incorrect transport arena layout.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds the foundations needed to support a virtqueue-based transport across host/guest, and wires that transport geometry into scratch layout and snapshot versioning so snapshots can faithfully restore or reinitialize transport state.

Changes:

Extend sandbox configuration, scratch layout, and snapshot config schema to include per-direction virtqueue geometry (queue sizes, buffer sizes, pool pages) and bump snapshot ABI/config versions.
Add host/guest virtqueue memory access primitives and guest transport context initialization during generic_init.
Introduce canonical ring image validation (incl. fuzzing) and refactor buffer/pool plumbing toward SlotPool + BufferLease/BufferMap.

File summaries

File	Description
src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs	Bump goldens version to v3.0
src/hyperlight_host/tests/integration_test.rs	Adjust OOM/abort expectations for transport
src/hyperlight_host/src/sandbox/uninitialized.rs	Update tests for new scratch sizing
src/hyperlight_host/src/sandbox/uninitialized_evolve.rs	Attach virtq on initialize path
src/hyperlight_host/src/sandbox/snapshot/tripwires.rs	Bump ABI and config media type expectations
src/hyperlight_host/src/sandbox/snapshot/mod.rs	Carry optional in-memory virtq snapshot
src/hyperlight_host/src/sandbox/snapshot/file/mod.rs	Persist/load transport layout fields; config v2 gating
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs	Add MT_CONFIG_V2; bump ABI to 3
src/hyperlight_host/src/sandbox/snapshot/file/config.rs	Add transport fields + normalization validation
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs	Tests for config v2 + transport round-trip
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Restore/attach virtq depending on NextAction
src/hyperlight_host/src/sandbox/config.rs	Add transport geometry to configuration
src/hyperlight_host/src/mem/virtq_mem.rs	Host `MemOps` for scratch + ring images
src/hyperlight_host/src/mem/shared_mem.rs	Add checked atomic load/store helpers
src/hyperlight_host/src/mem/mod.rs	Export virtq modules
src/hyperlight_host/src/mem/mgr.rs	Attach/restore/snapshot virtq in memory manager
src/hyperlight_host/src/mem/layout.rs	Place transport arena in fixed scratch prefix
src/hyperlight_guest/src/transport/mod.rs	Global guest transport context plumbing
src/hyperlight_guest/src/transport/mem.rs	Guest `MemOps` for scratch GVAs
src/hyperlight_guest/src/transport/context.rs	Guest virtq producers + pool construction
src/hyperlight_guest/src/prim_alloc.rs	Shared allocation limit helper + tests
src/hyperlight_guest/src/lib.rs	Export guest transport module
src/hyperlight_guest/src/layout.rs	Add scratch-top accessors for transport metadata
src/hyperlight_guest/src/error.rs	Convert VirtqError into guest error
src/hyperlight_guest/src/arch/amd64/prim_alloc.rs	Use scratch allocator limit helper
src/hyperlight_guest/src/arch/aarch64/prim_alloc.rs	Use scratch allocator limit helper
src/hyperlight_guest_bin/src/transport.rs	Guest transport initialization at boot
src/hyperlight_guest_bin/src/lib.rs	Call transport init during generic init
src/hyperlight_common/src/virtq/ring/fuzz.rs	QuickCheck fuzz for ring ops interleavings
src/hyperlight_common/src/virtq/ring/canonical.rs	Canonical packed ring image validation
src/hyperlight_common/src/virtq/pool/tests.rs	SlotPool unit tests
src/hyperlight_common/src/virtq/pool/slot.rs	SlotPool implementation
src/hyperlight_common/src/virtq/pool/fuzz.rs	QuickCheck fuzz for SlotPool invariants
src/hyperlight_common/src/virtq/msg.rs	Remove old virtq message header module
src/hyperlight_common/src/virtq/event.rs	Add canonical clear helper
src/hyperlight_common/src/virtq/desc.rs	Add descriptor table clear + base accessor
src/hyperlight_common/src/virtq/concurrency.rs	Update loom tests to new pool/buffer APIs
src/hyperlight_common/src/virtq/buffer.rs	Add Segments split/chunk APIs; BufferLease
src/hyperlight_common/src/virtq/access.rs	Add BufferMap interface for completed buffers
src/hyperlight_common/src/transport.rs	Shared message framing + external values
src/hyperlight_common/src/lib.rs	Export shared transport module
src/hyperlight_common/src/layout.rs	Scratch-top metadata struct + transport arena
src/hyperlight_common/src/arch/amd64/layout.rs	Make min_scratch_size checked/Option
src/hyperlight_common/src/arch/aarch64/layout.rs	Make min_scratch_size checked/Option
src/hyperlight_common/benches/virtq_api.rs	Update benchmarks to SlotPool naming
src/hyperlight_common/benches/common/mod.rs	Update bench harness to BufferMap/Lease
src/hyperlight_common/benches/buffer_pool.rs	Replace old pool benches with SlotPool
fuzz/README.md	Document added canonical image fuzzing
fuzz/fuzz_targets/virtq_packed_ring.rs	Fuzz canonical image validation paths
docs/snapshot-versioning.md	Update snapshot versioning docs for transport
docs/snapshot-oci-format.md	Update OCI snapshot format docs for v2 config
CHANGELOG.md	Note transport and ABI/schema changes

Review details

Files reviewed: 58/58 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Make pool restoration transactional. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Split and rename the pool implementations, add explicit lower/upper SlotPool regions. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Define deterministic producer and consumer reset behavior. Validate canonical events, descriptor chains, IDs, buffer policy, and unused descriptors. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Represent scratch bookkeeping with one repr(C) layout. Derive offsets and assert the host/guest ABI at compile time. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Use the first GPA of the reserved pages as an exclusive limit. Accept allocations ending at the limit and reject address overflow. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Define directional queue depths, buffer sizes, and pool page counts. Account for guest allocated rings and pools in minimum scratch calculations. Publish the transport contract through scratch-top metadata. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

This patch adds guest owned G2H and H2G rings and pools during guest initialization, and prefill H2G receive capacity. The guest then publishes their gpas through scratch metadata. The patch is also validates allocation order, scratch ownership, and canonical ring images before installing either host consumer. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Persist validated G2H and H2G ring images in running snapshots. Restore fixed transport allocations and install fresh host consumers before sandbox execution. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Allocate logical regions atomically Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Use queue_size instead of queue_depth Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

- mark producer's reset function as unsafe, - protect guest memops from reading/writing overlapping regions

jsturtevant · 2026-09-17T20:19:40Z

  pointer record selected via `index.json`. References one config and
  one layer by digest.
-* **config** (`application/vnd.hyperlight.snapshot.config.v1+json`). The
+* **config** (`application/vnd.hyperlight.snapshot.config.v2+json`). The


This is breaking right?

jsturtevant · 2026-09-17T20:24:50Z

-        // platforms) of guest stack, so add 20k more to get 24k
-        // total, and then add some more for the eagerly-copied page
-        // tables on amd64
+        const HEAP_SIZE: u64 = 128 * 1024;


is there a reason we increased the guest size here?

jsturtevant · 2026-09-17T20:28:18Z

+        assert!(sbox.mem_mgr.h2g_consumer.is_some());
+    }
+
    #[test]


I noticed alot of the tests in here had heap and scratch sized changed. What was the reasoning here?

andreiltd changed the title ~~virtq foundations~~ feat(virtq): add virtqueue transport foundations Sep 3, 2026

andreiltd changed the title ~~feat(virtq): add virtqueue transport foundations~~ feat: add virtqueue transport foundations Sep 3, 2026

andreiltd added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. area/fuzzing Fuzzing related test failures created from weekly CI labels Sep 3, 2026

andreiltd force-pushed the virtq-foundations branch from 7d903d0 to 1464929 Compare September 5, 2026 09:02

andreiltd added the regen-goldens Regenerate snapshot golden fixtures label Sep 5, 2026

andreiltd force-pushed the virtq-foundations branch from 4b50e87 to b06bf40 Compare September 8, 2026 14:41

andreiltd marked this pull request as ready for review September 8, 2026 14:41

Copilot AI lite review requested due to automatic review settings September 8, 2026 14:41

andreiltd requested review from danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners September 8, 2026 14:41

andreiltd added the ready-for-review PR is ready for (re-)review label Sep 8, 2026

Copilot started reviewing on behalf of andreiltd September 8, 2026 14:47 View session

Copilot AI reviewed Sep 8, 2026

View reviewed changes

Comment thread src/hyperlight_common/src/layout.rs

andreiltd force-pushed the virtq-foundations branch from 90f4c99 to 05913f7 Compare September 8, 2026 17:03

Base automatically changed from external-bytes to main September 9, 2026 14:15

andreiltd force-pushed the virtq-foundations branch 3 times, most recently from f2367eb to 5ccd23b Compare September 11, 2026 14:33

andreiltd added 24 commits September 16, 2026 19:09

refactor(virtq): remove reset api and harden allocation rollback

cf03a3e

Make pool restoration transactional. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): add tiered fixed slot allocation

317d073

Split and rename the pool implementations, add explicit lower/upper SlotPool regions. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): add canonical packed ring images

1eb9b4c

Define deterministic producer and consumer reset behavior. Validate canonical events, descriptor chains, IDs, buffer policy, and unused descriptors. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

refactor(layout): model scratch-top metadata

197a682

Represent scratch bookkeeping with one repr(C) layout. Derive offsets and assert the host/guest ABI at compile time. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix(guest): correct scratch allocator boundary

22e7083

Use the first GPA of the reserved pages as an exclusive limit. Accept allocations ending at the limit and reject address overflow. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): define virtq transport metadata

508d124

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): implement host side memory access

b0c9700

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(snapshot): preserve canonical virtq state

6ec6d01

Persist validated G2H and H2G ring images in running snapshots. Restore fixed transport allocations and install fresh host consumers before sandbox execution. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): optimize inflight bookkeeping memory

24e5bb8

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): use grouped allocation for virtq chains

672d2b6

Allocate logical regions atomically Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

refactor(virtq) adjust virtio terminology

afe3afc

Use queue_size instead of queue_depth Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: size foundation memory tests

6e0d086

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: add missing license headers

9b5f70c

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

feat(virtq): remove run pool

301c77e

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: inline allocation_exceeds_limit calls

dad0769

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: align up default scratch size to 16K pages

cadc366

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: validate queue types at construction

6e812a1

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

refactor: cleanup queue layout types

8fb02e9

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: typos

8b73c1f

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: adjust scratch size for fuzzing

1f9ce21

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

refactor: simplify host virtqueue access and snapshot admission

f004c35

Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

fix: address code review

ff9cd1a

- mark producer's reset function as unsafe, - protect guest memops from reading/writing overlapping regions

andreiltd force-pushed the virtq-foundations branch from 99f6ac2 to ff9cd1a Compare September 16, 2026 17:09

ludfjig mentioned this pull request Sep 16, 2026

feat: support guest log level overrides for initialized snapshots #1833

Merged

ludfjig approved these changes Sep 17, 2026

View reviewed changes

jsturtevant reviewed Sep 17, 2026

View reviewed changes

Navigation Menu

feat: add virtqueue transport foundations - #1793

feat: add virtqueue transport foundations#1793
andreiltd wants to merge 25 commits into
mainfrom
virtq-foundations

andreiltd commented Sep 3, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

jsturtevant Sep 17, 2026

Uh oh!

jsturtevant Sep 17, 2026

Uh oh!

jsturtevant Sep 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andreiltd commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

jsturtevant Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

jsturtevant Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

jsturtevant Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andreiltd commented Sep 3, 2026 •

edited

Loading