Skip to content

feat(ipc): request-id envelopes on every frame; responses in completion order - #25196

Open
charlielye wants to merge 1 commit into
cl/ipc-reactor-disconnect-cleanupfrom
cl/ipc-envelope-ids
Open

feat(ipc): request-id envelopes on every frame; responses in completion order#25196
charlielye wants to merge 1 commit into
cl/ipc-reactor-disconnect-cleanupfrom
cl/ipc-envelope-ids

Conversation

@charlielye

@charlielye charlielye commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Stacked on #25150 (contains its commit until it merges; review the head commit only).

What

Every ipc-runtime frame now carries a client-assigned request id, and clients correlate responses by id instead of position:

[4-byte LE length][8-byte LE request id][payload]      (length counts id + payload)

The server echoes the id on the response and sends responses in completion order — the reactor's reorder stash, its per-connection sequence counters, and the drain_disconnected_clients() GC hook are all deleted. respond() pushes onto a plain completion queue; the reactor drains and sends. A slow request no longer delays a fast one's response on the same connection (previously the stash re-serialized release even though the wsdb scheduler completed reads concurrently).

Ids are per-connection, random-start (so a stale frame left in a recycled MPSC-SHM ring slot by a previous occupant can never pair with a live call — largely defusing the known SHM stale-ring follow-up), with 0 reserved for server-initiated frames.

Why

The TXE flake investigation (#25150) established that positional correlation makes any stale/mispaired frame a silent decode-roulette. Ids make the whole mispairing class structurally impossible and detectable: a frame that pairs with nothing is now a loud failAll (“response for unknown request id …”) instead of a console.warn — closing the original hardening ask.

How (ipc-runtime only — codegen and all generated packages untouched)

  • C++ interface: IpcServer::receive(client, uint64_t& id) / send(client, id, …); run() echoes internally. IpcClient gains explicit-id send/receive virtuals plus serial convenience overloads (send(data,…)/receive(t)) that auto-assign ids and verify the echo — keeping the generated C++ clients and every FFI binding source-compatible. Pipelined callers must use the explicit-id API (the reactor tests do).
  • Mismatched-id semantics are per-transport (IpcClient::may_have_stale_frames()): SHM rings persist across occupants, so a frame addressed to a previous occupant's id is an anticipated leftover — the serial receive releases it and keeps waiting for the real response, and the TS async client discards it with a warning. Over UDS there is no reuse, so a foreign id is a genuine desync and fails loudly (serial: close; TS: failAll).
  • Transports: socket server/client (3-part writes, id strip on read), SPSC + MPSC SHM rings (ring_send_msg/ring_receive_msg carry the id inside the ring message), napi async glue (call(id: bigint, buf), callback delivers (id, buffer)).
  • TS: UdsIpcClient and NapiShmAsyncClient hold Map<bigint, pending> (random-start ids); UdsIpcServer parses/echoes. IpcClientAsync.call() signature unchanged — consumers are untouched (CdbIpcServer wraps UdsIpcServer, so it rides along).
  • C ABI: ipc_server_receive/send gain the id parameter; the handler-loop APIs (ipc_server_run*) and the serial client calls are unchanged — so the rust and zig bindings need zero changes (verified: they bind only those surfaces).

Version-mismatch story

This is a wire-protocol break; binaries and TS packages must move in lockstep (they already ship as matched pairs). Mismatches fail fast with explicit messages, both directions: a frame shorter than the id field is rejected as “IPC protocol mismatch (envelope ids); update the peer binary/package” (client-side failAll, server-side disconnect + log). Labs-side note: the acvm/wsdb npm + toolchain artifacts pick this up on their next republish pinned to a commit containing this change; the rust transport there is the same FFI crate, so no code change — just the lockstep bump.

Verification

  • ipc_runtime_tests 21/21 release and 21/21 ASAN — including SerialClientSkipsStaleFrameFromPreviousOccupant (SHM: injected stale frame is skipped, real response delivered) and SerialClientClosesOnForeignFrame (UDS: same injection fails the call). Pipelined tests converted from FIFO-order assertions to id-pairing assertions (each request answered exactly once); the reactor tests now also assert responses arrive out of send order under reversed-sleep handlers — pinning the head-of-line-blocking removal — and the MPSC test keeps its lost-wake stall probe.
  • New TS tests: UDS unknown id → loud failAll; id-less (old-protocol) frame → explicit protocol-mismatch error; SHM async client discards a stale frame while resolving live calls (mock-addon unit tests, incl. out-of-order pairing). ipc-runtime/ts suite 18/18.
  • echo_example (generated clients over the new runtime, codegen untouched): package + reliability tests green over uds and shm, including the SyncApi.
  • Cross-stack: rebuilt aztec-wsdb + new TS client running the world-state suites — sigpipe blast-radius test, churn legs A/B/C/D with 60s soaks (~1.2M id-paired calls, ~1.4k mid-flight-disconnect churn rounds), and the existing pipelined-read correlation suite over both transports — all green.

@charlielye
charlielye changed the base branch from next to cl/ipc-reactor-disconnect-cleanup August 13, 2026 13:35
@charlielye
charlielye force-pushed the cl/ipc-envelope-ids branch from ae4c3b5 to 9f5af45 Compare August 14, 2026 10:06
@charlielye
charlielye force-pushed the cl/ipc-envelope-ids branch from 9f5af45 to 2efd4ae Compare August 14, 2026 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant